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ABSTRACT 

A general evaluation model and workshop activities to 
help law-related educatibn (LRE) project personnel develop and 
implement evaluation plans for their projects are provided. The model 
and activities,, however, are applicable to other areas of the social 
studies as well. The first section of the handbook presents a model 
for evaluating LRE projects, giving an overview pf the evaluation 
process and suggestions for planning ^ntS conducting an effective 
program evaluation. The secjond, and major, section includes specific 
activities that can be used to t'each the concepts and skills 
necessary for effective ^valuation of LRE programs. Detailed lesson 
plans aref provided on a variety of topics, including how to determine 
priorities, how to state objectives, how to use Likert scales to 
evaluate student attitudes, hoW to use interviews to collect teacher - 
and student datsr,^and how to analyze and report dpita. An annotated 
bibliography of general evaluation resources Concludes the handbook. 
Appendices contain sample agendas for evaluation workshops and a list 
of LRE projects that have developed instrumentation for use in their 
owh evaluations. This list is keyed'to the grade level/purposed for 
which* the instrumentation was developed. (RM) 
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PRE.FACE 

'I' - . , ' 

Evaluation ^Js an essential elein^^t of every social studies prograiji. 
DecreaseiJ' funding levels and increased emphasis on accountability make 
Xhis observat:^^ especially, true today. 

Yet many jiducators know little about effetitive evaluation* They 
are unaware o^'the range ol evaluation techniques that may be applicable 
to their programs, and they are unsure of how to use the" data they do ^ 
collect. Educators often find the literature in the field difficult to 
translate ir^o terras that staff and participants can apjily on a day-to-* 
day basis . 

/ In pr^penting * series of workshops to law-related education proj- 
ect personjifel nationwide, the stAff of the Law-Related Education Eval- 
uation Technical Assistance Project succeeded in raaking the transition 
from the esoteric to the, practical. They developed a general evaluation 
model and/workshoj^ activities that are easy for project staff to use 
(1) to im^^rove their own evaluation designs and use\evaluation results 
to iraprov^^' their prograras and (2) to help project participants, includ- 
ing te^tli^-rs, develop skills in .usijig a variety of evaluation tech- 
niques. Wliile the examples used in the raodel and activities are drawn 
from law-^^elated education, both tqols are e^Jually ap/licable to evalha- ^ 
tion of ot^her social studies prograras. * 

ThtiSj^/this publicatior^ will 6e raost directly applicable to J^hosc '^^ 
involved iri the growing law-related* education moyementT butvit will also 
be a usefuJL .source of raat^rials- t^haC^can be adapted to other uses*i It. 
i§ witli tria^ JLn^treTft' that ERIC/ChESS and SSEC CQoper^te in xhe publication 

"^f this handbook. - . . ' 



Jaraes E. Davis 

Associate Director, SSEC and ERIC/ChESS 
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<v.f .INTRODUCTION / 

* \ This handbook was originally designed to be ^ practical guide to 

help law-related education (LRE) project personnel develop and implement 

evaluation plans for their projects, but the principles, and guidance 

provided are equally applicable to other areas of the social studies as 

wi^ell. The first section of the handbook presents a model for evaluating 

LRE projects. This model provides an 'overview of the evaluation process 

and general suggestsSons^ for the question^ *and issues that need to be 

V ; 

ad<Jressed ,in planning and corvdjjcting'^n effective program evaluation. 

, The second section includes specific activities that a project 
director or other leader can^^se to t6ach the concepts and skills neces- 
sary for effective evaluation of LRfe prograpis to their staffs and to 
others involved in LR£ actij^itig.S^* While detailed lesson plans are . 
included to facilitate ready use^bf the activities, they can 'be adapted, 
revised, and used according to;the leader *s particular needs, stylei.and, 
time schedule. The wor'kshop^ materia Is can also be[. adapted to include 
examples drawn from other ar^eas of the social studi^e^, thus making them 
more broadly applicable. .Illustrations of how the'j activities might be 
use^ in dif f erent -workshf^ formats are presented irj Appendix A.'- ^ 

An"-wannotated bibi:f.ography concludes the handbook. This bibliography 
includes a bjrief lis^ of commercially available .resources that might 

represent a "core" evaluation . library* A longer list of applicable 

/ 

resources in the ^%1Q system is also provide^d. All the resources listed 
are general, father than focusing only upon l^w-related education. Thus, 



they will be of interest to educators in all facets of the social studies, 
field. Apperidix.B contains a list of LRE projects that have d^velgped ' 
instrumentation for use in their own evaluations. Tliis list is keyed to 
the grad^ level/purposes for which thil instrument;ation was developed, 
thus facilitating use of the list to identify sources of instrumentation 
that xaay be useful to a project addressing a sitailar audience/purpose. 
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. A MODEL FOR' EVALUATING LRE PROJECTS ' \ 

This program evaluation model, is a ^pragmatic effort to assist law- 
related education program personnel in more effectively evaluating their 
programs. A variety of appro5(^hes to evaluation can be taken — 
qualitative, quantitative, and^ goal-free to mention a few. The model 
^described here incorporates some aspects from many of these approaches, 
but it is certainly not the oAly way to evaluate an educational program. 
It is, however, a pragmatic/ approach to evaluation', * an approach ' that 
will he^ljJ project personnel conduct evaluations^ that 'provide results 
which can be us^d to makie programmatic improvement^ as well as to 
determine program impactyi This pragmatic model, assumes that the *pro- 
grams to be evaluated are based upon a needs assessment-and upon values, 
and policy decisions ^J^^^ arc reflected in their goals and objectives. 
Although these steps /have not been included in the model, it is 
important^, for projects that are just beginning to conduct a needs' 
assessment, malce policy decisions based upon explicit values, relate 
program goals to the needs statement, and derive specific objectives 
from important variables identified by the needs assessment. 

Figure 1 is a diagram of 'the model, whose steps are described 
below. « ' 

Identify/Clarify f^rojcct Goals . . ' 

The initial fetep in evaluating an ongoing LRE program is to iden- 
tify and clarify the program's goals. ' Goals are "statements, usually 
general and abstract, of desired states in human conditions and social 
environments."* They^ are general statements th^t should be closely 
related to the program rationale. Goals are an expression of the end 
conditions that the program see^s to achieve;^ they establish a direction 
f6r developing specific objectives that can .be niea&ured. 

Goals can be divid-ed into tv;o Broad categories-- -programmatic and 
instructional. Programmatic goaJs are broad,^ general statements of what 
acti^llties and organizational changes the program seeks to initiate. 



*Peter H. Rossi, Howard E. Freeman, and Sonia R. Wright, Evaluation: 
A Systematic Approach (BeverJy Hillsl Bage Publications, 1979), p/-5A. 



A MODEL FOR EVALbATING EDUCATIONAL PROGIWIS 



IDENTIFY/CLARIFX 
PROJECT GOALS 




IDENTIFY/CLARIFY 
OBJECTIVES RELATED 
TO GOALS 



REPORt FINDINGS >/ USE ANALYSIS TC 
MAKE JUDGMENTS AND PROVIDE FORMATIVE 
and' SUMMATIVE /eEDBAcR THAT CAN BE 
USED IN MAKING PROGRAMMATIC DECISIONS 



ANALYZE DATA 



COLLECT DATA. 
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IDENTIFY ACTIVITIES NEEDE.t) 

TO achieve;' programmatic 

INSTRUCTX'ONAL OBJECTIVES' 



DETERMINE PURPOSE AND ^ 
SCOPE '^F THE EVALUATION \ 



DEVELpP THE RESEARCH DESIGN • 

1. Identify types of evidence 
Identify criteria for sue- 
/" cess 

/Identify location of evi~ 
/ dence \ 
' Identify an appropriate 
research Resign arid data 
collection techniques 

5. Consider control group 
issues 

6. Select/ develop instru- 
ments 

Consider reliability/ 
validity issues 
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They can describe final states of organizations, processes, or aefcivi- 
ties that will be conducted. Instructional goals describe in broad 
terms what program participants are to know-> feel, or do as a 'result of 
their participation. For example, a programmatic goal might be to 
establish a law-related education component *next year in all of the 
ninth-grade civics classes in the Fall River School District, An 
instructional goal might be for students? in the ninth-grade ' civics 
classes to learn about the American legal system. , 

* Identifying and clarifying 'goals involves several processes. 
* First, it is importdnt to determine that the goals listed in progi^am 
descriptions are actually the goals pursued' by all project staff iind 
^^hat the listed goals have the same meaning for everyone. Oft^n a dis- 
cussion of these issues reveals that project staff have dif f erent^inter- 
pretations of their program's goals. In this event it is import?ant to 
establish consensus regarding goals, • / 

A second part of the identification and clarification process is 
determining whether' a goal can be translated into measurable objectives'. 
This requires considering what a goal means in terms of the c^haracter- 
istics of the program o^ student learning; Measuring achievement of a 
programmatic goaL is generally easier than measuring achievement^ of an 
-/instructional goal. If the evaluator knows what material was picsonted 
,i-n ninth-grade civics clashes one year,, determining whether more law- 
related issues were addressed th,e succeeding year would be relatively 
simple. Measuring learning about the law, on the other hand, requires 
that someone define exactly what it is about the law that students are 
to learn. This type of assessment i$> much mor^e difficult. 

Another consideration. is ^ the appropriateness of the goals. Are 

they appropriate given the financial and personnel resources available 

* « 

to the project? Do they seem attainable in light of cifrrent research on 
the topic? 

Once goals have been written lo be congruent with these considera- 
tions,^ the evaluator is ready to determine which goals are 'programlnatic 
and which are instructional. This distinction is impo]^lant; becausq the 
two types of goals often require different evaluation techniques, focus- 
ing upon each type individually will facilitate tl|c evaluation. The 
distinction is often fuzzy, however, primarily because programmatic 
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gdals are the stepping-stones jthrough which inst.ructional goals are ^ 
achieved. This relationship is the source of much of the confusion and 
ambiguity in classifying goals. 

For practical, purposes,, instructional goals for educational set- 
tings can.be defined as those'goals that can Ultimately be measured in 
-terms of ^impact upon the knowledge, skills, and attitudes or a student 
group. Any group thiat is expected to gain in, knowledge , skills,,^ or | 
attitude_^ as a result of some treatment can be called a student group. 
Far example, if teachers are to learn new classroom skills during a 
workshop, they a-rb the student groiap with regard to the learning ot . 
those skills. * • * 

When instructionsll goals are elaborated as objectives, their 
achievement can be assessed by measuring jjhanges in student behavior. 
Programmatic goals, on the other hand, are associated with the activi- 
ties which indirectly inj[lu6nce students and contribute to changes in ' 
knovledge, skills; or attitudes. Programmatic goals are generally . • 

evaluated by no'ting the behavior of program personnel. Many program- 
matic goals are reduced to objj^ctives that can be measured in two 
ways—by recording whether an * event occurred and by 'measuring the 
quality of the processes related to the event. 

Identify/Clarify Project Qb.jectives ' 

After separating programmatic from instructional goals, the eval- 
uator must begin to identify. a|ji, clarify the objectives associated with 
each goal'^ Evaluators have defined programmatic and instructional 
objectives in a variety of ways; the descriptions below indicate how 
these terms Were defined in developing thi£^^ model. Programmatic objec- 
tives refer to successful completion of a series of related activities 
th^t are necessary to achieve the broad program gdals. These objectives 
focus Hpon activities that do not directly involve student - learning. 
For example, recruiting teachers selecting materials, and budgeting h 
funds would be p'rog'rammat ic object ives*^ as'sociated with the goal of ' 
-initiating LRE in ninth-grade civics classps throughout the district. 

Instructional objectives specify what learners will be able to do ^ 
after participating in the' program that they could not do before. 
Instructional objectives can be written with varying degrees'of speci- - 

- - • " ^ ' • 
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ficity. At 'tKe lowest acpeptable l*evel, they focus on describing h » 
behavior that will demonstrate that learning has occurred,^ For example^ 
"Stiiden'ts^ \^ili be, al^'l-e to describe six rol^es in a criminal court case" 
is a loosely written objective^ The,mosf specific instructional objec- 
tives^ specif ylavels and conditions df performance. For example, a very 
rigorous instructional objective' that might" be .associated with the goal 
of*havitig students learn about the criminal law 'system is, "When , given j|t 
list of s^ix roles in a criminal court case, 85 ^percent of ^the studM^s 
will be able to write a brief description of each role, men'^tioning 90\ 
percent of' the characteristics of each." * 
Instructional objectives should:' 

1. Focus \ipon student learning. 

2. Identify an observable behavior that de^nonstrates learning. 

3. Relate directly to a broad instructional goal. 

. A. Allow flexibility in instructional materials and procedures. 

5. 3e appropriate for the developmental level. of the audienqe. 

6. Be* relevant to the needs of the audience. 

7. Be categerized as cognitive, affective, or skill objectives. 

8. Be rea'^istic. Changes that are not likely to- result from 
participation in, the program should not be included as objec- 
tives, c 

Evaluators sTiould review each objective carefully to determine that it^ 
meets these criteria. Objectives^ which do not should be rewritten. 
Before the objectives are rewritten, the project staff should b*e 
involved in precisely defining the objectives. Staff involvement will' 
help to establish consensus about objectives and will provide the eval- 
xuator with a comprehensive overview*of the project. * 

Identify Activit;ies Necessary to Achieve Programmatic and Instructional 
Obj ectives 

After identifying and clarifying objectives, the evaluator and 
program staff should identify the . activities neaded to achieve^ each 
objective. For programmatic objectives, activities are defined as those 
actions which must be successfully .completed in order to achieve pro- 
grammatic objectives. These activities generally do not directly ^ 
involve students. For example, sending training announcements to 
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principals. might be one activity associated with the program objective 
of recruiting teachers for an LRE training progranv. Instr^ictional 
activities are those actions intended to produce changes in the^ knowl- 
edge, skills, or attitudes of students; they do involve studenps 
directly. -For example, having students participate in a mock trial 
might be an instructional activity associated witfi^ tlie objective of 
having students be *able to list six roles involved in a criminal court 

m 

case. 

Identification of -programmatic and instructional activities is 
important for several reasons. First/ it may l^lp program personnel 
discover flaws in their program design: Perhaps some necessary steps 
^have been overlooked* Second, it p/Tovides the evaluators with a handy 
checklist for assessing project progress in the area of programmatic 
objectives. This is especially true il the activities are time-lined. 
It also provides the evaluators with target act ivit ies ' aboXit which to , 
collect data. The evaluator thus gains some idea about vh^r e ac tivities 
related to various aspects of the program will occur. This knowledge is 
helpful as the evaluator begins to consider specific kinds of data 

n.eeded to determine how well the program is meeting its objectives. 

^ * 

Determine Purpose and Scope of Evaluation 

Next, the evaluator should meet with project personnel and evalua- 
tion sponsors in order to determine the purpose of the cvaluatiori; 
About what areas are they most concerned? About what topics do th^y 
want data? What do they intend to do with the findings? Jo what audi- 
ences will the findings be presented and what will those audiences do 
with the findings? 

It is essential to determine in advance who expects what from the 
evaluation and what they intend to do with what they learn-. This infor- 
mation is useful in a variety of ways. First, it helps the evaluator 
focus upon the most need.ed data. For example, s'chool boai^d members may 
be interested in the effect of LRE upon attendance because many states 
base their reimbursement rates to^ districts upon the bumber of students 
present per day. If effects on attendance were only a' minor project 
objective and resources for evaluation were scarce, an evaluator might 
not collect attendance data without advance knowledge that the board was 
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interested in it. Second, this information i^ helpful in selecting data 
collectimr techniques. Some audiences will dhly be impressed by "hard" ' 
statistical data while others are more inti&rested in the detail and,, 
richness provided by qualitative techniques. Identification of audi- 
ences and their preferences may influence the types of data collection 
and analysis procecj^res adbpted , 

Third, many eval^iators hope to provide information that will be 
useful to program personnel and assist them in making p^rogrammatic deci- 
'Sions. By identifying whkt program persons want to know and what they 
intend to do with data, the evaluator can focus the evaluation on areas 
that will address those needs. Another benefit of identifying audiences 
and their needs .is that this knowledge is useful when selecting appro- 
priate means for presenting findings, A general narrative summary with 
some anecdotal evidence may be appropriate for community groups, whil^.a 
sophisticated, statistical treatment may be more appropriate for a 
federal funding agency. , « 

-In addition, this phase provides another opportunity to establish 
congruence between the goals and objectives pursued by program personnel 
and those evaluated. Determining the purpose and scope of the evalua- 
tion will help to assure that the evaluation addresses actual needs and 
that the results wilL*be used. 

This phase of the evaluation should a4so be used to determine when 
and how formative feedback will be provided. Formative feedback is 
information about how well a project is operating that is provided to 
project personnel periodically during the lifetime of the project. This 
is opposed to summative feedback, which generally assesses a program's 
impact at the end of the project. It is important to establish in 
advance how evaluation findings will be provided to the project ^taff so 
that they can use the information to make pTogrammatic decisions. 

Developing the Research Design 

The initial steps in^ the evaluation model have had a linear rela- 
tionship. The next step, however, can be best understood as a collec- 
tion of issues and questions which must be addressed before conducting 
the evaluation. In many instances these issues and questiotfs are inter- 
related, and an answer to one may make answers to others obvious. The 



issues and questions do not^need ^ be addressed in order, but a 

Response to each should be developed by the conclusion of this step. 

During, this^ phase, the evaluator Should consider the following seven 

issues: ' ' t , * 

' \ > ' 

1 . What types of evideiice mj/fehJ: be ^used to determine if an objec- 
tive has been "achieved ? An essential step in the development of a 
research design is the identification of what types of information are 
needed* to^ assess the achievement bf obj ettive^s If the objectives have 
been written rigorously, specifying behaviors', conditions for measure- 
ments, and criteria' for supcess, this question does not need to be* 
addressed again. However^ if the project staff has written looser 
ob-jeortives, the evaluator(> must consider ^What evidence would indicate 
that| ah objective has been ^achieved. For example, student responses to 
a request to describe six roles in a ^criminal court case would be evi- 
dence that would suggest the degree to which an instructional objective 
had been achieved. The presence of teachers at a training meeting might 
be evidence to indicate achievement^ of a f)rogrammatic objective. ^ 

Identifying types of evidence that would be useful in determining 
how* well objec^:ives have been achieved provides an evaluator with a 
target for data collection efforts* Knowing what types of evidence are 
needed helps the evaluator determine what kinds of data should b^ col- 
lected. Qften, a variety of types. of evidence tan be identified *for 
each objective* Ideally, information about all of these types of evi- 
dence would be collected. If data from several sources is collected by^ 
, usingf different collection techniques, triangulation can ^ be -•used . 
Trian^ulation is the process 6t comparing data from dif f erent " sour^ces^ 
but relating to the same topics.' If the data from different sources are 
congruent, stronger conclusions can be drawn than if the data were 
derived from only one scoirce. 

Realistically, however, financial and time limitations often make 
collection of data from multiple' sources impossible. The evaluator must 
choxjse from the various^ types of evidence identified th(5se which will 
provide the best evidence while being capable of being collected within 
the project '^s financial and tjime constraints. 

In considering the kinds of evidence to be used, the evaluator must 
'recognize that there are different levels of evidence. For example, if 
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an evaluator wants to determine if students know fiow to file a claim in 
small claims court, a very direct assessment would be to ask students to 
file claims and observe what they-- do, ^less* direct way would be for 
the evalufator to ask students to indicate how much they know about filing 
^a claim. The least direct way might be to ask g^teacher how much the 
students know. Generally, the more direct the l.evel of evidence, the 
better data it will provide, 

' 2, What, are the criteria for success for each type of evidence 

7 : 

identified ? It is important for instructional and programmatic objec- 
tives that involve a quality assessment to Include criteria for success% 
For example, if an instructional objective is to have students describe 
six roles in a criminal court trial , ^'the number of correct response*s 
that constitutes success must be identified. Must all students describe 
all six roles in order for the objectiv^e to be achieved? Has the objec- 
tive been achieved if 1^ percent of the students correctly describe five 

•roles? Similarly-, if 60 percent of participating teachers indicate that 
they are using new techniques in their classrooms, has a programmatic 
objective been achieved? These questions can only be answered after the 
evaluator determines precise , criteria for measuring successful achieve- 
ment of objectives in light of each type of evidence identified as beings 

.related. to each objective. Criteria can be reasonably developed in two ^ 

way^. First, a review .of .the relevant literature will indicate what has 

been achieved previously and suggest appropriate expectations . for the 

objective being considered* However, if, relevant research data are 

scant or unavailable, discussing appropriate criteria for su*ccess with 

persons knowledgeable abbut the target population will b*e the more 

feasible approach, ^ 

3, Where can the desired evidence most likely be found ? This step 

% * ' 

ijsimply involves determining where one could find the type of evidence 

identified previously. For example, program files would be a •likely 
location for copies of ;:he roster' of teachers attending an LRE training 
program. Supporting evidence in the form of workshop evaluations, 
budgets, and an^agenda might alsjo be found there. Sources of informa- 
tion about changes in teachers' classroom behavior might include the 
teacher, studients, other teachers, and supervisory pers^onnel. By 
identifying sources of evidence, the' evaluator adds more structure to 
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the evaluation effort. The evaiuator can. then identify sites that will 
requirel permission be'fore data can be gathered, and .the process of 
secuting necessary permis^sions can begin. The evaiuator can also group 
by sourc^p the kinds of evidence needed. This grouping may be helpful if 
the efvaliiator wants to collect all of the information from one source at 
one time.l Identifying sources of evidence thus assists the evaiuator in* 
•managing the evaluation process. 

4. What type of research design is most appropriate and what are 
the fnost appropriate methods for collecting evidence ? A variety of 
, factors should be considered before these cfuestion^ are answered. At • 
the very least, the evaiuator should review the purposes of --the evalua-* 
tlon and the audiences that will be receiving evaluatJLon- data. In 
instances where, having a great deal of information alfout program' 
processes at a limited nupiber of sites is desirable, a qualitative case 
.study design may^ be most appropriate. ^ Case study designs attempt to 
provicle a holistic description of a site. They seek to 'describe a set- 
ting or organization as the "natives" see .it\ Descrj,ption of patterns - 
and regularities provide^ the basis for making conclusions and judgnvents. 

In instances where concern f opuses upon program impact, a quantita- * 
tive scientific or quasi-scientific research design may be best. Such 
designs attempt to determine how a small number of independent variables 
influence a small number of dependent variables. This approach tries to 

V 

determine how a s{)ecific treatment influences a target group by comparing 

a small number of variables in the target group to the same variables m 

• * 

a similar group. ' ' s 

For most evaluations, a blend of quantitative and -qualitative tech-r 
niques is appropriate. Although both techniques can be used in both 
summative and formatJLve evaluations, quantitative techniques a%e more 
commonly used to measure impact upon narrow and precisely defined vari- 
ables while qualitative methods are most commonly used when it is diffi- 
cult -to^ isolate variables or when, the. evaluation is primarily concerned - 
with processes. Qualitative methods include such techniques- aa open- 
ended interviews, open-ended questionnaires, observational records, 
collected field clocumencs, and logs written by participants. ^ Quai)tita-, 
tive techniques are those ^ approaches wiiichv seek to assign nutnerical 
values to the data assemblfed. This approach uses techniques such as 

o . 12'; , ~ \ • 
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forced-choice questionnaires, structured observatjLons , checklists, 
structured interviews, and formal tests. Data in these forms >can be 
converted into numerical form, permitting simple but useful statistical 
treatments as well as more sophisticated treatitlents involving computers. 

One critical concern iti selecting a research design and data col- 
lection strategies is cost. Generally, assembling good qualitative data 
requires extensive investment in »person^hours , which is costly. On the 
other hand, qualitative data is often the most useful in making program- 
mat it: decisions and adjustments. Quantitative daj:a may be processed by 
hand,^ but more sophisticated treatments may require -a considerable conK 
l5uter budget. The eyaluator must therefore consider how to allocate 
evaluation resources in order -to gather the most useful data. 

5. » Should control groups be used and if so, what kind ? The issues 
of whether or not to use a control group and if so, how to select it, 
can be rather complicated. Ideally, a control group is a group that is 
identical to 'the experimental group in all respects except for the vari- 
able or 'Characteristic bfeipg tested. Comparing the results obtained for 
the experimental group with those of ^ the control group increases .the^ 
evaluator'/s confidence in any conclusion that changes in the experimental 
group wer^ due to the special treatment of them and not to some other 
variable.' In other words, the ideal control group would tell the eval- 
uator what happeife if the experimental treatment is not used. 

The ide§l is, of course, difficult or impossible to achieve in^ 
reality. Ait example can illustrate some of the problem^. Suppose you 
are evaluating the effects on student knowledge of a new LRB curriculum 
in an American government class. Your experin\ental clas3 is the class, 
receiving instruction^: in this new material. You give the students a 
test of their knowledge of the material at the end of the semester and 
fi^d that they score 80 percent correct on the average. Vrtaat should you 
use as the control group? 

You could pick anothei; class in American government that is rot 
receiving instruction in the new ^.RE curriculumy But is this control 
class really identical t6 the experimental class in all respects except 
the experimental treatment? Are their intelligence and motivation levels 
identical? 'Can you^e sure they are not learning the material in axiother 
form? rs""their initial knowledge level of the material identical to the^ 



experimental class? Are "they as' homogeneous a group as the experimental 
♦ ' <i 

group?, * • , 

To the extent that yqu don t know whether the experimental and 
control groups are identical, it will be difficult or even^ impossible to 
interpret your results. In the above example, in which the experimental 

giToup scored 80 percent, how would you interpret the, results if the 

■f i 

control group scored 85 percent? Does this mean the new LRE currict^lum 
was not effective? Ndt necessarily. Perhaps the control* group was more 
intelligent or generally iilore knowledgeable. ' Perhaps a few individual^ 
had some unusual experiences? which brought^the class average up.-. In 
short, if the experimental and control groups are ,upt absolutely iden- 
tical (which they never can be), prol?lems in interpreting the results 
may arise. (Note that ,the same problems of interpretation would /occur 
whether the experimental group scores higher c)t lower than t+ie controls.) 

In practice, the trick is .to try -to identify the major variables 
that could make a difference, and look for a cont;rol group as similar as 
possible on these variables. Sometimes, advanced ^ statistical techniques 
can be used -to factor out known differences; this is a job for an expert 
statistician.' Also, the evaluator must make' sure th^re is no inadvertent 
treatment of the, control group that could confound the results. 

One of the most frequently used methods is to use the same students 
as ^their own control — usually by administering a pretest as^well a§ a 
posttest. There are pitfalls t^ere, too, however. For exampleV simply 
taking attest twice can result in improved scores the second time. Or, 
unexpected or unknown events occurring between the pre- and posttfests 
can affect the posttest scores. Or some special situation, unknown to 
the evaluator, may have affected pe"?*f ormance on eirfver test — pfrhaps the 
pretest^Nwas given late in the afternoon and ^the"^osJC test early in the 
moyning. 

' • The above discussion is not meant to discourage the use of control 
grdups, but» rather to point out tlie complexities inherent in their use 
and to encourage the evaluator to think through the implications df any 
particular control gi?oup selection in interpreting the results. ^ 

6. . What instruments are available for collecting evidence' or what 
instruments must be constructed ? Two common mistakes occur in addressing 
this issue: to completely adopt existing instruments developed for 
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similar programs or to immediately begin to write items and develop 
instruments. Both of. these practices; should be avoided. Using an 
existing ' evaluation instrument without modification is fraught with 
, danger's. Often an' instrument designed for use with'Xgne project wil] 
'give very poor results when used with .another ^project because tlie 
instrument addresses the specific objectives ' of the first project and is 
nQt appropriate for any project., with different objectives. Hbwever, 
individual items from existing instruments n^ay b*e useful in .assessing 
achievement of objectives for other programs. This approach may be 
particularly advisable if there are national norms for. the existing 
items and the testing populations are similar. ^ ' ' 

It is equally mistaken to immediately sta^ developing instruments 
without surveying the field to determi/e what is available. Although. it 
is unlikely that existing instruments will be totally' 'satisfactory with- 
out modification, it, is possible that existing instrurtents can be adapted 
or individual items selected from them, thus saving time and perhaps 
providing items with' established reliability and validity. 

Thus, the evaluator should begin the process of -instrument construc- 
tion by, reviewing what is available and assessing its applicability in 
light of project objectives. Items that can be used without change or 
with modifications should be adopted. The evaluator should then develop 
items to assess achievement of objectives f.^or which no existing items 
were found. The project staff should critique the items in order to 
establish that they address project objectives and arc appropriate ior ^ 
the target audience. Instruments should also be field teate'd. The' * 
evaluator should enter the field test with specific questions in mind. 
Some of the more commonly asked questions are: How long does it take to 
administer this instrument? Is the instrument appropriate to the target 
audience?^^! s the instrument reliable? Is it valid? 

7. What levels of , reliability and validity are acceptable ? Reli- 
ability is the degree to which an instrument produces Identical results 
when administered Onde^ identical conditions. Without a reliable instru- 
ment,. Comparing tests administered to similar graups' at different times 
is difficult." Validity refers to the degree to which an instrument 
actually measures what it purports to treasure. ^ Obviously^ if an instru- 
ment does not do this, it cannot provide an accurate assessment of 
achievement of program objectives. 

y 



High degrees of reliability and validity are clearly desirable.' 
The more reliable and valid instruments are, the greater the amount of 
confidence one can have in conclusions based upon analysis of data 
•gathered with them. Often an evaluation section within a school dis- 
trict or a graduate university class can help to determine test and item, 
reliability and validity. 

Collect Data 

After corvsidering the questions and issues related to developing 
the research design, the evaluator should collect and organize data. 
The data collection effort should be carefully monitored using a time 
line. The evaluator, in cooperation with program staff, should^ deter- 
mine when data about each indicator will , be collected. This information 
should be charted and used to guide the evaluation schedule. Data col- 
lected should "be listed on the chart. This process will allow the eval- 
uator to identify what data is needed at any pointy in the process, help- 
ing to avoid the problem of realizing near the end of an evaluation that 
data about three indicators are missing. 

Maily systems can be used to organize data. For the most part, it 
is a matter of personal preference which is used, but any system should 
meet certain requirements. First, as data is collected, it' should be. 
filed in a logical manner that will facilitate easy retrieval and identi- 
fication of missdng information. In addition, the data collection and 
filing system should guarantee the privacy of those involved. If pos- 
sible*, maintaining multiple copies of all data is advisable; thus, if 
data is lost from one location, it can be reconstructed; Finally, 
access to data Should be limited to those who need it. This policy helps 
to control data loss and protects the privacy of subjects. 

V Analyze Data ^ , . . , . 

The next step in The evaluation process is data analysis. For / 
qualitative evaluations this process occurs simultaneously H^ith data^ 
collection. As data is gathered, the evaluator studies it for trends 
and patterns. As these emerge, data is filed according to these emerging 
findings and %he evaluator identifies methods of testing the findings by 
looking for additional support and contradiction^. 




" Quantitative data analysis, however, gene^lly occurs at specified 

points in the programmatic prooe-srs^^ usually ^ near the end^f the evalua- 
tion process •^^^^.^xe-' and posttest data can usually be computer analyzed ~ 
at .Ibhe-^same tim'e, although exceptionally large samples may necessitate 
the'><p^ocessing of pretest data shortly after collection. 

Voften the analysis of quantitative data involves the use . 9f infer- 
entiai^* statistics, statistical tests that provide information regardii^g 
the likelihood that results could have occurred , by chance, rather than 
because^^of program activities. "Jhe use of inferential statistics^^ 
nccessar^ whenever the evaluator wants tb make an inf erent^e about a 
t^tal population from the data^ obtained from just a sampled Common 
technique^' include chi-square, t-tests> and analysis of variance. 
Assistance ^-n computing *these statistics can often be obtained from a 
local testijig or evaluatiou'persop dr from a nearby college. 

Evaluators should try to synthesize the findings from the quanti- 
tative and qua^litative co'^iponents df the, evaluation. Frequently, quali- 

tative data, can, be used to explain quantitative findings . This holistic 

* * . f * " " 

^p^)roac|i to evaluation prfJVides a more satisfactory evaluation of pro- 

^^ram prpcesses^ arid impact, .fhan dichotomizing findings along quantitative- 

y^jqualitati^^e lineS"* • , ' ' \ 

^ Report Findings ' ' ' - 

. The ^next task 'is to report the findings. In presenting bindings, 
the evalujitor must \ often deal with two -conflicting requirements: 
(bj. p;resenting the dita in a way that the audience will understand in 
the ^rqount of time^ they are willing to give to the task and (2) p-resent- 

' ing ''j^^ie data accurately, without masking its subtleties and qualtfica- 
tioii$..and without the distortions that can accompany simplification and 
summeti^ization of data. Here, a creative use- of descriptive statistics 
is in order. These statistical^ techniques provide ways to describe a . 
group and to commurilcate its essentiaj. characteristics quickly and 
easily > Common techniques inclufde the use -of charts and graphs, and 
such statistics ajs^^means, medians, standard deviations, and correlation 
coefficients. 

r 

In the case of formative evaluation, reporting should occur formally 
several times during the life of the evaluation. Informal reporting 
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will probably occur more -frequently . Formal feedback usually takes one 
of two forms; fl) a written report provided to- programvpersonnel or 
(2) verbal communication with the project staff. Both forms are highly 
useful. A written record provides the sjtafC with an unchanging document 
to which all can turn to identify successes and failures, thus reducdng 
ambiguity, and confusion about^ what the evaluator found. On the ot^ljer 
hand, verbal communications can provide the flexibility needed to address 
the concerns of the project staff. This approach allows the evaluator 
to respond to the unique concerns of various program personnel,^ but it 
has the weakness of being subject to misinterpretation and skewed percep- 
tions, Therefore, both written and. verbal feedback ,are suggested. In 
preparing formative feedback in any format, the evaluator should indicate, 
the degree to which goals and objectives have been achieved, make judg- 
raents,* and offer suggestions and recommendations. 

Summative assessment almost always assumes the form of a final 
evaluation ^ep6rt . This report should be written with the intended 
audience in mind. If there have been additions to the audience origf- 
nally identified, the new audiences' needs and interests should be 
addressed. Unless the evaluator has other guidelines, the most logical 
means for organiEing the final report is to report the findings by goal 
and objective. Each section shpuld indicate the degree to which the 
subject goal and objectives were achieved. It is also important to 
remember that a primary purpose of most evaluations is to provide infor- 
mation" for use in making progr'ammatic decisions. Therefore, in a sum- 
mative^, report i\\e evaluator may want to include a section of recommenda- 
tions and "suggestions for program personnel, as well as for the funding 
agency; if that is appropriate. 
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A FACILITATOR'S GUIDE FOR PROVIDING LRE EVALUATION ASSISTANCE 

This section of the handbook contains 14 workshop .activities for 
u^e with project staff and others involved *in LRE activities. The 
activities are designed to help those persons understand the steps in 
the evaluation model described in the previous section and to develop 
the skills needed to implement those steps. 

The handbook activities are adaptable to a variety of situations 
and curriculum or program obj ectives . We recognize that workshops in 
which LRE project directors or other leaders have enough time to present 
alitor even most of the activities will be rare. In addition, project 
staff or participating teachers and administrators may already have ade- 
quate understanding of some steps in the evaluation process and/or may 
not need to be familiar with all the steps. We have therefore designed 
the activities so that leaders can select those activities which seem 
most appropriate, arranging them in the order that best meets their 
needs. Thus, one leader may use "selected activities in a single work- 
shop, another may use nearly all of the activities in a workshop series, 
and a third may use only one or two activities to teach staff about a 
particular evaluation technique. We also encourage educators working in 
ot|ier areas of the social studies to vary the examples used in the 

activities—now drawn exclusively from LRE — to, make them applicable to 
« 

the particular content with which their participants are working. 

Two aids to planning a workshop agenda are presented. The chart oji 
the next page provides a brief overview of each activity, the length of 
time required, the step of the model to which the activity is related, 
and a list of the materials needed. All handouts and transparencies 
^^listed as being required are provided as black-line masters following^ 
^the activity in which they are used; copies will need to be made. The 
facilitator wiJ 1 need to supply any underlined materials. Appendix A 
illustrates how the- activities can be used in. different workshop 
configurations. 
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GotU Cor UE 

St stint Object Ivei 

OrgtiiUlng ind KeClnlni^ UE ^lU 
snd Object Ivei 

Cvtluatlon Techniques ind Lav 
Reltttd Educttlon (^leitloni 

A Variety of Cvtltiatlon Technique! 

Indlcttort •nd Techniques 



'Using Llkert Scsles to Evsluste 
Student Attitudes 

Using Hultlple'^Cholce Tests to ^ 
' Evaluate Student Usmlng 

Using Intervlevfl to' Collect Tescher 
and' Student Dst« 

f 

Using Observstlons to Evsluste 
Kole risyi a 



Collecting Dsts snd Organltlng Files 6 



" ReUte4 Step 
in ftodtT^ 

Hooe; Introductory 

All 

1 

2 

l»-5 



Deacrlptlon 
JUnk order, discussion 
Lecture/discussion 

Card sort, discussion 
Wbrkiheet, discussion 
Uorksheet, discussion 




tl>c Iteq'd 
30*i»5 alnutes 
20-30 alnutes 

I hour 
i-l>l hours 
I'-l^ hours 



irslnstonilng, worksheet, discussion 30-<»5 alnutes 



Reading, discussion 



<»5'60 Minutes 



fndlvldujil or snail-group planning 2-3 hours 



/ 

Ubrktheets, discussion. Item 
developaent 

Ubrksheets, discussion, Itea 
developawnt 



3 hours 



2 hours 



Anslyxlng snd Reporting Dsts: 
Selecting the Appro|lrlste 
Descriptive Ststlstlcs' 



7-8 



Role plsyi Instruwient developiwent, 2*3 hours 
'discussion 

Bralnstonilng, discussion, role plsy 1-1^ hours 



Resdlng, discussion 1^ Minutes 

Lecture/discussion, works)ieets, 3 hours 

ssiall-grDup prbblea solving 



IkterlsU Needed ** 

Hsndout 1^ 

Trsnsparency 1, Handout 2, overhead 
projector 

LRE goals csrds, Handout 3 
Handouts <»-7 
Hsndouts ^ snd 8 



Handouts * snd 10, newsprint snd Marking 
pens or chalkbosrd snd chalk 

Handout 11 

Hsndouts 2 and 12, oewspiflnt snd aarkint 
£e|« 

Hsndouts 13-17. > newsprlnt snd —rklng 
pens or chslkbosrd snd chslk 

Hsndouta 18-21 



Handouts 22-2*, newsprint snd —tklng 
pens or chslkbosrd snd chslk 

Handouts 25-27, newsprint snd Marking 
pens or chslkbosrd snd chslk, bsg of 
powdered sugar, shoe, signs Indies ting 
vsrious locstlons In s house* csrd" 
bosrd or re si television ^ 

Hsndout 28 , 

Hsndouts 29-32*, newsprint, markjnt P^na, 



*Steps In fiodel} 

l"ldentify/ctsrlfy project gosls ^ 

2— Hfatlfy/clsTlfy objectives relsted to gosls 

3— Identify activities needed to achieve prog r^swis tic snd Instructional objectives ^ 

4- *d«tenRine purpose*snd scope of the evslustloi^ ^ 

5- -d«velop the resesrch deaign ^ * 
•-collect dsta ' , ' 

7- -»analyre data ij V 

8- -report findings 

S€% pp. 3-17 for s datslled sigplsnatloo of theae^ steps. 

♦♦AH handouts tnd trsnsparancles sre provided ss blsck-Une «stets following the sctlvltles; copies will need t6 be 
Ih« faclUtstor will need to avpply any underlined Materlala. 



^de. 



27 



ACTIVITY 1: 
DETERMINING PRIORITIES 

« 

Introduction ; Edujcators, especially teachers, do not usually^ place a 
very high priority on evaluation. The purpose of this rank^rder activ- 
ity is to clarify that evaluation is one of many important activities in 
which LRE teachers must engage. The act,ivity will set the tone for your 
asking the participants to place more value on evaluation, but not to 

drop everything to develop and administer pretests and posttests, 

I 

\ 

abject ive's ; Participants will be able to — 

1, Identify those LRE activities upon which they place , the most ' 
importance, *• 

'2. State the main reasons for their particular priority rankings, 
3. State several reasons why it is important not to forget about 
evaluation in conducting LRE, projects. 

Time ; 30-A5 minutes 

Materia^ls Needed ; Handout '1 

Procedure ; > 

1. Begin by stating that teachers of , law-related education, engage 
in many important activities and that the purpose of this exercise is t^ 
clarify, how they do or would prioriti^ those activities that compete 
for their attention, r 

2: Distribute a copy of Handout f, /'Determining Priorities," to 
each participant, - asking therfT^o complete the rank-prder activity as 
^Indicated in the directions, ** ^ 

3. Then ask participants to (1) write down the main reasons fo¥ 
their first and se;cond choices and (2) select one of the activities that 
they did not place in the top five and write down the maia reason that , 
it did not receive a high ranking, 

A, Ask participants to share their particular rankings ^and rea-. 
sons. If the group includes more than 25 or 30 participants, you may 
want thei^ to discuss theiir Rankings in small groups before initiating a 
full-group discussion, 

ft 
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5. Summarize' the major results of the discussion by identifying 
those activities that generally received high priority and those that 
received^lx)!^ prioriCy . Most groups will rank the class learning activi- 
ties highest (e.gr^,^ field trip, police as speaker, simulation) ^and the 
evaluation activities lowest (e.g., behavioral objectives, pretest/post- 
test, and semantic dif ierential)-*^^ 

6, Discuss and identify the mailr treasons why evaluation so often 
receives, a low ranking. Reasons usually include lack of training in the 
area, the idea that teaching is more important than testings ^nd evalua- 
•tion's time-consuming nature, 

?• State that similar factors, including the pressure of daily 
management, may prevent many LRE projects at the national, state, and 
local levels from conducting evaluation activities as thoroughly as they 
would lil^ to, 

8, State or brainstorm several reasons why it is important for 
LRE educators to plaile some priority on evaluation. For example: 

— Evaluation can help projects operate more efficiently and effec- 
tively, 

--Evaluation can help in efforts to disseminate and implement LRE 
in other schdols and classes by providing evidence about what students 
learn. 

— Evaluation can, suggest changes that should be made to improve 

project operations or materials, 

— -.Evaluation may be needed in order to attain additional funding, 
— Evaluation is needed to demonstrate that proj^dt objectiyes have 

been achieved. 

9, Conclude by stating that the remaining exercises and materials 
in the workshop will help participants learn how to conduct evaluation 
activities efficiently for the benefit of the entire LRE program. 
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Handout 1 ^ 1 of 1 

(Activity 1) - - ^^"^ 

DETERMINING PRIORmis 

Directions ; You are a social studies tea^iher committed to teaching about 
law and our legal system. The district requires you to record two grad'es 
per week, so you must make up and grade ^ome tests, quizzes, and papprs. 
But you are free to decide how you spendjthe rest pf your time. Listed 
below are ni^e activities that would b^good" to do with and for your 
students. Even taking work home in t]^e evenings, however, you can prob- 
ably only get five of these 'things done. Which would you do? Rank your 
choices from'l to 5 with 1 being your top priority:. 



plan a field trip to^the county courthouse to see a trial. Pre- 
paring for the trip will require organising transportation, check- 
ing with teachers and . administrators to avoid creating conflicts 
in the school calendar, getting parental permission and super- 
visory support, working with the judge to maximize learning, etc. 

Develop, and use a simulation of a crimfe and the apprehension and 
trial of an accused person, A robbery of the school store cpuld 
be staged, followed by an investigation, filing of charges, and a 
trial. \ 

Write specific behavioral objectives ^for the unit, being careful 
to include verbs that describe observable behaviors, the condi- . 
tions under which the accomplishment -of the objectives will be 
measured, and the , criterion for successful completion of each ^ 
elective, > ^' * ' 

Take pre/posttest data from the last unit that the class studied 
and relate each item on the test to some as{)ect'of the curriculum 
materials and class , activities used. Then make inferences from, 
the test data about which activities were successful in promoting 
. learning. 

Vi^it the home of a student who seems withdrawn and lonely at^ 
-school, . ' : 

Do some background readinf in the development of law through 
history. This is an area you were exposed to only superficially 
in your undergraduate teacher preparation. 

Construct an attitude survey to determine if your unit changed 
the way students feel about 'of f icers of the law. «' ' • 

Spend time in the public library with one student trying to, find 
information abou^^hat lawyers dc/. This student is bright and 
has shown a keen^nterest in becoming a lawyer, but was unable to 
find anything on lawyers in the school lihrary. 

Arrange with the local police derpartment to bri^g a police 
officer into the classrdom on a regular basis to' talk informally,* 
answer questions, and just be around the students. . 



Adapted from Robert W. Rlchburg, "Classroom Assessment in a Law ^nd 
Humanities Elementary School Program,'' in Daring to Dream; Law agd the 
Humanities for Elementary Schools , Lynda Carl Falkepsteln^ and Charlotte 
Anderson, eds, (Chicago: American Bar Association, 1980), *p. 276, Used 
by permission. ' . ' • ' 



ACTIVITY 2: 
THE EVALUATION PROCESS -IN LR£ 



Introduction : Most people J.n educatioi; have a very narrow view of eval- 
uation. Teachers often equate it with testing for the sole purpose' of 
grading. For LR^ projects, evaluatioa usually involves having partici- 
^tjits say that you did a gopd job. The purpose of this lecture/discus-' 
sion activity is 'to present^ a model of the entir^-^<igScess of evaluation 
for law-related education. The. intent is to convey .the idea thajt eval- 
uation is a Systematic, multi-step process that includes many procedures 
directly related ^o organizing and conducting a project effectively. 

<^ 

Objectives ;* Participants will be able to — 

1. Identify the majo-r steps in the presented model of evaluation 
for LRE. ^ * ^ * 

2. - State the key questions that need to be addressed for each 
step in the process. 

3. State examples from^LRE that relate to each step in the proc- 
ess. " 

Time: 20-30 minutes 

Materials Needed : Transparency 1, Handout 2, overhead projector ^ 
Procedure : 1 

, 1.^^ introduce this activity by icating that many different models . 
of the evaluation process have been developed* and used in education. 
Indicate that the model you are going to present has been developed and 
used /specifically for evaluating LR*E projects, thopgh it could be appli- 
cable to other/areas. 




Project Transjparenay 1, "An Evaluatioi^ Model for Law-Related 
Education Projects." You may also want to „distri^te a copy of the 
chart to each participant. Identify the major questions that need to be 
addressed in each step, illustrating the discussion with examples from 
LRE. The description of the .model in the preceding section of this hand 
book provides the information you will need for this t^resentation. (You 
might also assign that section aS k reading for participant^.) 



3. Answer any questions participants have concerning the model 
and its application to LRE. 

A, Distribute" copies -of Handout 2, ''LRE' Project Hanageinent and 
Evaluation Form," to the participants. Note that this two-page form is 
a practical tool for relating five of the elements in the evaluation 
model — project goals, project objectives, project activities, indica- 
tors, and ^evaluation . techniques . Participants will use thd form later 
in\he workshop to do just that. Also point out that the first three 
steps o| the process (the three items on the first page of the form) 
should be implemented by a project regardless of any intention to form- 
ulate and conduct an evaluation. Stress *that full application .of .this 
evaluation process will help ensure effective project management, 

5. Conclude by stating that this .evaluation modei represents the 
ideal way Vo conduct eval^uation in Tfaw-related education but that in the 
real world we must .often settle for something less than the ideal. 
Explain that -the I'eraaining activities ino the workshop will focus on 
specific aspects ot the model in order to provide participants with the 
knowledge and skills needed to implement the various steps in the model. 
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Transparency 1 
(Activity 2) 



1 of 1 



A MODEL FOR EV7SHJATING EDUCATIONAL PROGRAMS 



IDENTIFY/CLARIFY 
PROJECT GOALS 



IDENTIFY/CLARIFY 
OBJECTIVES RELATED 
- TO GOALS 



IDENTIFY ACTIVITIES NEEDED 
TO ACHIEVE PROGRAMMATIC 4 
INSTRUCTIONAL OBJECTIVES 



REPORT FINDINGS: USE ANALYSIS TO 
SIAKE JUDGMENTS AND PROVIDE FORMATIVE 
AND SUMMATIVE FEEDBACK THAT CAN BE 
USED IN MAKING PROGRAMMATIC DECISIONS 



DETERMINE PURPOSE AND 
SCOPE OF THE EVALUATION 



ANALYZE DATA,,' 
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COLLECT DATA 



DEVELOP THE RESEARCH DESIGN 

1. Identify types of evidence 

2. Identifjr criteria for suc- 
cess 

^ ______ - — ^ 

Identify loeatlorr of evl- 
"dence 

Identify an appropriate 
research design and data 
collection, techniques 

5. Consider control group 
issues 

6. Select/develop instru- 
ments 

7. Consider reliability/ 
validity Issues 



34 



Handout 2 
(Activity 2) 



PROJECT GOAI.: 



LRE PROJECT MANAGEMENT' 
AND EVALUATION FORM 



OBJECTIVES 



ACTIVITIES 



ERJC , 



3 



Handotut'2 2 of 2 

c 

LRE PROJECT MANAGEMENT 
AND EVALUATION FORK' 



INDICATORS 



3 



ERIC 



EVALUATION TECHNIQUES 
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ACTIVITY 3: L ^: 

. ' * GOALS FOR LRE 

Introduction ; Development of evaluation plans for LRE projects begins 
wi\h identification and refinement of the project goals. While differ- 
ent LRE projects can and do have different goals,, certain -criteria 
should be kept in mind when clarifying goals. ^ The purpose of this card- 
sort activity is to hqlp participants apply these criteria to a -set of 
LRE goals, thus enabling them to do the same with tfte specific goals of 
their projects. . ,^ 

Objectj-veg : Participants will be able to — ^ \ 

1. Distinguish between goals that are" appropriate and attainable 
for a short-term LREvproject and those that are not. 

2. Explain the difference between programmatic and instructional 
goals. 

^Z. Classify a set of LRE.goals as either programmatic or instruc- 
tional. ^ . . 

Time : 1 hour ' . 

Materials Needed : LRE goals cards, Handout 3 

t* * 

Procedures : ' • ^ ' 

1. Introduce the -activity by asking participants to brainstorm^ 
typical goals for LRE projects. This should result in a variety of 
goals, some fairly narrow in scope and others very abroad. State that 
the purpose of this activi'ty is to help participants examine and refine 
project goals so that they are clear guideposts for later work in speci- 
fying objectiyts, activities, and evaluation procedures. 

2. Arrange the participants in groups of four to five members and 
distribute a set of LRE goals cards to each group. Tell participants to 
determine which of the 15 goals in the packet are appropriate and attain- 
able for" a ^ne- to two-year LRE project and which are not. The groups 
should then sort the cards into two piles on this basis. 




3. Ask each group *to state which goals they thought' were "not 
appropriate and attainable and why. While some disagreement may occur, 
the goals that are stated so broadly as to b^ unattainable in a one- to 
two-year project are go^ls 1, 5, 7, 10, and H. (Th6se are, of cour'se, 
not absolute answers. The size of the project .apd the community and 
schools involved are important factors in determining ability to reach 
those goals.) Other statements may need more specification at the objec- 
tives level, but could be attainable and are appropriate. Emphasize' 
that, unlike objectives, goals cati be stated in general, nonbehavioral 
terms, but that to be useful their scope must be within reason given the 
project's resources and time. 

4. Distribute a copy of Handout 3, ''Defining LRE Go^ls," to each 
participant and discuss and clarify the definition of a goal and the 
difference between programmatic and instructional goals. ^ 

5. Tell ^ach group ^o sort the ten appropriate goals ,into two 
categories — prdg^rammatic and instructional. r 

6. Ask each group to state which' goals they ^classified as program- 
"matic ancT which as Instructienal. .Check these with our classifications: 

programmatic — 3,*4, 9,* 12, 13, and 15; instructiopral — 2, 6,. 8, and 11. 
There° may be some disagreement about goals 6 and 9, both of which 
involve teachers. State that teachers ^are usually an intermediate 
target; students' are the ultimate /target group. For our purposes, 
however since 6 is -stated as an outcome and 9 as *a process, they should 
be classified accordingly. ^ , 

7. Conclude by stating tjiat participants .will have an, opportunity 
to apply these criteria to their own project goals after an activity 
focusing' on objectives for LRE. , 
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LRE Goals Cards 


1 of 4 
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GOAL , 1 • 


GQAL 2 


■ . i 

Citizens in Florida will be 


Students ViLL understand the 


MbRE KNOWLEDGEABLE ABOUT 


BASIC CONCEPTS PERTAINING TO 


CIVIL-LAWIN THE STATE. 


THE LAW AND THE LEGAL SYSTEM.. 
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GOAL' 3 . 


COAL A 


. TwF PRn.lFPT Will DEVELOP AND 


The metropolitan schools will 


p\/a[ MATP a PPOrF^^ FOR IM" 


RECEIVE HELP fOR THEIR EXIST- 


PLEMENTING LRE THAT CAN BE 


ING LRE PROGRAMS, 


REPLICATED IN SCHOOLS ACROSS 




Tilt? ^rMIMTD\/ 

THE COUNTRY 1 
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■ LRE 'Goals Cards 
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GOAL 5 

Junior high students will 
act as more effective and 
responsible citizens when 
tKey become adults. 


Coal 6 .. _ 

Teachers will demonstrate 
knowledge of new lre content 
and instructional strategies. 

t t 

* 




« 




* 
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GOAL 7 


GOAL 8 




fli 

f 


Vandalism in the New Yobk^ 
'subv/ays will be reduced • 

* 

i 

t 


StUDEN^TS in THE PROJECT WILL 
LEARN ABOUT THE LAl^ AND LEGAL 

SYSTEM IN Virginia. 

$ 


> 
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LRE Coals Cards 
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. GOAL 9 

Teachers will be trained to 
use law-related. education- 
materialseffectively in 
their classrooms. 


GOAL 10 

The project will develop and 
institutionalize an lre pro- 
gram for the city schools. 

t _ 

« 

f 
#■ 


COAL 11 

• 

s i 

Students will develop more- 
positive ATTITUDES ABOUT 
THE law/ POLICE^ AND THE 
JUSTICE SYSTEM."^ 

» 


GOAL 12 

The project wIll establish j 
a resource center for 
teachers in the district. _ 

r 

• 
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LRE Ooals Cards 
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GO'aL 13 


GOAL \l\ 


The proje'ct wiLi^ infuse LRE 


The rate of juvenile delin- 


INTO THE SOCIAL STUDIES 


quency IN THE CITY WILL BE * 


CURRICULUM AT THE ELEMENTARY 


REDUCED. 


LEVEL. 




♦ 

1 ^ 
\ * 


0 

« 


i 

GOA{. 15 




The project wTll involve 




COMMUNITY GROli^^S IN THE 




I-MPLEMEr^TATION OF LRE I,N ' ■ 




THE SCHOOLS. 




•% 
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Handout 3 * 1 of 1 

(Activity 3) - . * 



» DEFINING LRE GOALS 

Goals : Project goals are "statements, usually cgeneral and -abstract , 

of desired states in human conditions and social 

environments/'* Goals express the end condit^ions that the 
*prograifi seeks to achieve. 



Programmatic Goals ; Programmatic g6als are ,broad, general statepients of 
^the activities and organizational changes the program seeks to 
initiate. ' . ' * 

Example The project will estafblisn a law-related education component 

next year in all of t/he ninth-grade civics classes in Ehe*Fall 

River School Distri^e^t. ^ ^ • 

^ * 



^Instructional Goals: Instructional goals describe in broad terms what 

^ <. V* 

program participants are to knpw, vfeel, oi; do as a result of 
their participation. " 

E;^ample': Students in 'ninth-g.rade civics classes* will learn about the 
American legal system. • ' * 



1 



*From Peter lU Rossi, Howard £• Freeman, and Sonia R. Wright, 
Evaluation; A Systematic Approach (Beverly ^ills: Sage Publications, 
1979), p. 34. 



ACTIVITY A: 
STATING BRE OBJECTIVES 

Introduction : Specif icaticui of project objectives is one of the most 

iraportan't^ steps in the LRE evaluation process. While all agree that 

well-Stated i^bjectives are^a key to effective evaluation, views on what 

constitutes a well-stated objective vary. Rather than adding another 

view to^this debate, this activity helps participants work through sojne 

—of the main factQj;;^ to be considered in specifying LRE objectives-* The 

activity is intended to help participants decide on acceptable levels of 

objectives for their projects, thereby setting the stage for the follow- 

ing activity, 
«> 

Objectives ; Participants will be able to — * . ^ 

1/ Identify arrange of acceptable and unacceptable LRE object^es 
in terras of their levels of s|fecif icity • 

2, State several important characteristics of a well-stated objec- 
tive. 

3, ^Write LRE objectives that fit a defined range ©f acceptability. 
Time: l-l^c^hours 



Materials Needed: *■ Handouts 4-7 



Procedure : 



1. Introduce this activity by stating. that while goals can be 
very broad and general statements of desired outcomes or accomplishments, 
objectives must be stated more specifically, ^focusing more on actual 
desired behaviors. Continue the introductary remarks by making the 
following points: 

a. How to specify objectives for instructional and programmatic 
purposes has 'been the focus of Considerable work and discussion among, 
educators. ' There has also been considerable disagreement regarding what 
constitutes an acceptable statement of objectiveS^. One issue is whether 
a given objective is an important one for LRE programs. This issue, 
whil^ important, is not an evaluation issue and therefore is not dealt 
with here. 
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Another area o£ disagreement Is the level of specificity 
required in a statement of objectives,. Some educators demand more rigor 
than others. For example, for some people, "Studeijjts will be able to 
identify the main roles in a criminal -court case" is an acceptable objec- 
tive. Those who ojpt for strict behavioral objectives would not find 
this statement acceptable. They would prefer a statement similar to the 
following:" "Eighty-ffve percent of the students in the program will be 
able to list at least six rol^s in^l cr:j.minal court case." 

c. We do not. believe that it 'is e^Sy or necessarily useful to 
draw an absolute distinction betweem acceptaMe and unacceptable objec- 
tives. Reasonable people have reasonable d:^fferences 6t opihion on this 
issue. We do believe, however, that objectives for LRE programs and 
projects must be stated specifically enough *^to provide the evaluator 
with some guidance in how the objective will be evaluated or measured 
and in how to teH when the objective has been achieved. 

d. This activity involves participants in working with LRE obj^pc-- 
•tyives so that t^ key points and issues are illuminated and clarified, 
thus enabling them to determine how specifically their own objectives 
should be stated. 

2. Distribute copies of Handout A, "LRE Objective's: A Clarifying 

« 

Exercise," and ask participants to complete the 'exercise according to 
the directions. 

3. After participants have completed the worksheet, debrief by 
asking the follov/ing questions: Whi^ objective was the first one rated 
as acceptable? Why? Did you rate any of the statements as U/A? Did 
anyone reach, the point of overkill? Where? Most -.participants will 
begin rating the. statements acceptable at number 3, although they may 
reconsider once they have heard the later ones. Point out the key 
factors that need 'to b^ addressed for this objective to be helpful for 
evaluation purposes: how many teachers, how many role plays, the content 
of the role plays, and the time period within which they should be com-^ 
pleted. Such other information as the number of students and their roles 
is helpful but may not be necessary, depending upon the nature and scope 
of the project and its evaluation plans. 

A. Point'-out that this example has dealt with the extent and 
specificity of the ^quantity of outcome, not with its quality. In many 



case«» specifying the level of quality desired is also advisable.' 
Quality can also be stated at various levels of specificity. 

5. Distribute copies of Handout 5, "LRE Objectives: Specifying 
Quality," and have participants circle the number of the phrase they 
believe^ defines an acceptable level of specificity. Discuss their rea- 
sons for these choices. 

6; Point out .that^ if you know what instruinen1;s you are agoing to 
use, you can get very specific in formulating your objectives. If not,, 
then fhis level of specificity will need to be dealt with when instru- 
ments are developed or data is analyzed. 

7. Distribute copies of Handout 6, "Action Verbs for Instructional 
Objectives," pointin|p^out the importance of using action verbs in stat- 
ing*' objectives specifically. . This is a necessary but not sufficient 
condition for a well-stated objective. Have participants scan these' 
lists; discuss any questions they have. 

8. Distribute copies of Handout 7a, "Writing- Well-Stated LRE 
Objectives." Ask participants to complete the task as indicated on the 
directions. Tell them they may use the list of action verbs in rewriting 
the objectives. Have participants share their answers. There ^y be 
several different statements of each objective that are acceptable, 
depending upon the criteria of acceptability that each participant has « 
established. Have participants compare these statements to those on the 
first handout to determine what range of specificity they reflect. 

9. Conclude by summarizing that well-stated objectives: 
— FocUs on actual behaviors. 

IS 

— Provide enough specificity to help evaluators decide what to do. 

— Provide enough specificity to help program people know when objsec- 

f - -J. 

tives have been attained.. 

— Are directly related to a goal. 



Handout 4 
(Activity 4) 
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LRE OBJECTIVES,: A CLARIFYING EXERCISE 

Directions : Shown below are eight statements of an LRE objective stated 
at increasing levels of specificity • As you read each statement, decide 
whether you think the statement is acceptable in terms of specificity. 
Use the following to record your responses: U = Unacceptable; A =t 
Acceptable; 0 = Overkill (more specificity than you want). 

If you are not, sure, use the t^o letters that correspond to the two 
responses that you are debating abouf (e.g., U/A) . If you change your 
mind about one statement after reading a later one , cross out your 
original respoij|^e and add your new one. 

♦ 

' The Statements 

GOAL : .Train 20 secondary social studies teachers to use various LRE 
strategies in their social studies classes. 

OBJECXIVES: 

1. LRE project teachers will 'conduct role plays. 



2. All LRE project teachers^will conduct role plays in their socia 
studies qlasses. 

3. All LRE project teachers will (conduct at least three role plays 
in their social studies classes. 

C J 

4. All LRE project teachers will conduct at least three r-ole plays 
in their social studies classes within two months after the 
workshop. 

5. All LRE project teachers will conduct at least three role piays 
on search and seizure and police powers In their social studies 
classes within, two months after the workshop. 

6. All LRE project teachers will conduct at least three role plays 
on search and seizure and police powers, eaelv. taking at least 
20 minutes, in their social studies classes within two months 
of the workshop. 

7. All LRE project teachers will conducX^.at -^least^t:hree"role'*^plays 
pn search_and--seizuTe~'an"d"^o^ powers from the CRLSMFT materi 

• als, each, taking at least 20 minutes, in at least two" of their 
social stsUdies classes within two months of the workshop. 

8. All LRE project teachers will conduct at least three roj.e plays 
*on search and seizure and police powers from the CRLSMFT materi 

als, each taking at least 20 minutes, in at least two of^ their 
social studies classes within two months of xhe workshop, so 
that at least 50 percent of the students in these classes have 
played at leMt once the role of policeman, judge, or criminal. 



Handout 5 
(Activity* 4) 
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LRE OBJFXTIVES: SPECIFYING QUALITY 



These phrases could be added to the statements of objectives from the 
previous handout to add a ''quality** dimension* 



1. will successfully conduct 

2. will successfully conduct as determined by student surveys and tests 

3. will successfully conduct in terms of student interest, involvement, 
and knowledge gain as determined by student surveys and tests 

A. will successfully conduct, as indicated by a moderate to high level* 
of student interest, involvement, and knowledge gain a^ determined 
by student >^urveys and tests'" 

5. '^will successfully conduct as indicated by a moderate J:o .high level 

of interest , involvement , and knowledge gain format least 80 percent 
of the students in the LRE program as determined bx s.tudent surveys 
and tests ' , 

V 

6, will successfully conduct as indicated by »at least a 3.5 score on 
the "LRE Interest and Involvement Inventory (IIJ)** and at least a 
5,0 gain score on the- "LRE Police Role Play Knowledge Test" for at 
least 80 percent of the students in the' LRE program 
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Handout 6 ^ s ' ^ . ^ 1 of 1 

(Activity '4) - ' v ' ' 

ACTION VERBS * , ^ 

FOR , . ^ • 

INSTRUCTIONAL OBJECTIVES ^ 

0 * 

KNOWLEDGE: cites", defines, describes, identifies, iisfs,, matches, names, 
points out , recalls, recognizes, relates; remembers, repeats",^ 
rephrases, reports, states, tells, writes . 

COGNITIVE PROCESSfeS: > ^ / ^ • * 

Classifying : arranges, associates , catalogs, distinguishes, groups^ . 
labels, orders, organizes, outlines, places, rearranges, reorders^ 
• sorts , structures . 

Interpreting : composes, converts, defines, enacts, explains, expresses, 
illustrates, interprets, paraphrases, presents, renders, restates, 
simulates, states in own words', summarizes, transforms, translates 

Comparing : , cites, descr^ibes, detects, differentiates, discriminates, 
distinguishes, expresses, points out \ 

*'* 

Generalizing : composes, constructs, derived, develops , * expresses'; forms, 
formulates , genera t es, produces, proposes , relates ^ 

Inferring : deduces , develops, derives, draws, extends, extrapolates, 
fills in, formulates, generates, presents/' proposes 

Analyzing ; analyzes, breaks down, detects, d^-f f erentiates, divi-dcfs,^ 

examines, experiments, expresses, extracts, identifies, illustrates, 
inspect-s, inventories, lists, outlines, points out, questions, 
relates , separates 

Synthesizing : ^ assembles, composes, constructs, combines, creates, 

depicts, derives, designs, develops, devises, expresses, formulates, 
ilrlustrates', integrates, makes, organizes, prepares, plans, pro- 
duces, puts, together, proposes, synthesizes 

Evaluating : appraises, argues, assesses, chooses, critic^izes, decides^, 
describes, evaluates, ^explains, grades, judges, justifies, measures^ 
ranks, rates, rejects, scores, states worth of, validates, weighs 

AFFECTIVE PROCESSES: advocat^, acclaims, approves^ believes in, 

cl^oses, defends, demonstrates, opposes, , praises, prefers, reacts, 
pd^tively or^ negatively^^ toward, recommendSj;^ rej$ut^\ selects, ' ^ 
sypports 

SKILLS: carriers ouf, completes, constructs, draws, executes, interprets, 
locates, measures, performs, puts into practice, shows, translates 
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Handout.' 7^ 
(Activity 4) 
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WRITING WELL-STATED LRE OBJECTIVES 



Directions : Rewrite each objective that you believe needs more clarifi- 
cation and specificity, ^ 



1, r Students will be able to explain how to use a small ^:lairas court, 



Students will increase their citizenship participation in the 
school. 



3, Teachers will be able to identify the case study approach as a main 
feature of the Street Law materials. 



Participating students will be more likely than control students to 
express "^the belief that all persons should be treated equally by 
the law, ' . ' ^ 



Tftis project will influence the implementation of LRE in^Vhe city^s 
schools. 



6. Each of the ten participating teachers will us"e at least five pf 
tlie 15 LRE units during the .course of the year. 



?• Ninety percent of the students ^will learn eighty percent the 
basic concepts in the CRF materials, " . ' 



o2 



ACTIVITY 5: - - 
* ORGANIZING A^JD REFINING 
LRE GOALS AND ^.OBJECTIVES, 

V ,1b, 

Introduction ; Participant.s now apply the concepts and skills learned iit 
the previous activities to the goals and objectives of their particular 
projept^. Using 'the ''LRE Project Management and Evaluation Form" they 
"organize their goals , ^objectives, dnd project activities to prepare for 
-T)lanning their evaluation procedures. 

♦ * ' ,^ 
Objectives ; ^Participants will be ^ble to — < • . 

1. Writ§. their project goals u^ing ,the principles^ taught in pre- 
vious activities. 

2.. .Write their objectives using the principles taught in previous 
.activities. ^ ^ « 

3. Organise their lists of project goals and objectives using a 
form. , . ^ 

4. • Wr-ite project 'activities as,sociated with tjieir specific objec- 
tives. . , , 

Time ; 1-1^ hours ^ ^ 

Materials Needed : Hahdouts 2 and 8 ' * 

V- 

Prodedure; " . . 

1. ' Introduce this activity by stating that the time has 'come to 
apply the principles for writing good <LRE goals and objectiv,es. 

2. Distribute copies of Handout 8» "LRE 'Project Management and 
Evaluation Form: ' Page 1 Illustration," and explain that this is the 
format they will use to specify and organize their* project goals ^nd 
bbjectives. State fhat this form helps to tie together goals, objec- 
tives, and activities, and, later, othej aspects of the evaluation plan. 

3. Tell the participants to coinp!|.ete page 1 of the form (distrib- 
ute;^ in Activity 2) for their project^ following the ex-ample provided. 
Circulate among the participants to provide help as needed and to ensure 
that pveryone is on the right track. If tijne is limited, have each 

; 3.1 
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participant focus on only one goal and a limited/number of objectives. 
They can then complete the process following the workshop, 

•^4. Conclude by asking participants to share their observations 
aismt doing this task. State that they are now ready to pove on to the 
heart of the evaluation process. 



» 4\ 



Handout 8 
(Activity 5) 
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PROJECT GOAL (Programmatic) 



LPvE PROJECT MANAGEMENT 

, AND EVALUATION FORM: ^ ' . • , ' 

PAGE 1 ILLUSTRATION ' * * \ . 

A group of social studies teachers will be trained, to use law-related 
education materials and attivities effectiv.ely in their classrooms. 



OBJ^TIVES 



ACTIVITIES 



.TARGET DATE 



r.O Patticipating' teachers will be ^hle' to 
identify tbe main features of St^reet 
Law, CRF, and LFS materjals. 



. 0 Partitipatirig teachers , will te^ch at 
least one activity from each of these 
three 'sets of materials. ^ 



'3.0 Part icipating' teachers will select 

specific activities and materials and' 
identify where in their existing 
/ courses they will be taught. 



1 



1.1 An intensive two-week training 
workshop .for 30 social studies 
teachers will be held in which 
• materials and activities from 
the Street Law, Constitutional 
•Rights, and L^w in ^ Free Society 
projects will be demonstrate^ and 
examined. 

r 

2.1 - During the two-week workshop-* ■ 
teachers will be trainjed to 
use A variety of activities » 
from tbese materials and will 
»v be pnrovided opportutiities to 

practice using these activities. 

3.1 At the conclusion of the work-* 
shop teachers will list those , 
material's and activitie sjj^t hey 
will teacH in their existing 
cou'rses and will include these 
in an outline of those 'courses . 



8/14-8/27/81 



8/14-8/27/81 



•8/27/81 
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ACTIVITY 6: * 
EVALUATION TECHNIQUES AND * ^ , 

LAW-RELATED EDUCATION QUESTIONS' 

Introduction : Many evaluation techniques can be used to assess LRE 
programs, depending upon the specific objectives and questions of the 
project. This activ'i>^^ is intendeds to introduce some of these techniques 
and to help participant^ think about their most appropriate applications , 
in law-related education. 

Objectives : Participants will be' able to — 

!• • List several different qualitative and quantitative evaluation 
techniques. 

2. Select the techniques most appropriate for specific LRE evalua- 
tion questions^ 

3. Identify factors that need to be considered in selecting appro- 
priate evaluation techniques. 

Time : 30-45 minutes * . 

Materials Needed ; Handouts 9 and 10, newsprin't and marking pens or 
chalkboard and chalk 



Procedure: 

> 



!• Intro^^e the activity by asking participants to brainstorm a 
list of evaluation techniques. Post these on the board. 

2. Explain the difference between qualitative and quantitative 
techniques, using the description provided in the model. 

3. Take each technique listed on the board and ask the partici- 
pants to classify it as^ qualitative or quantitative. Make sure that 
their answers match those provided in the model.' 

4. State that each type of technique has certain advantages and 
disadvantages depending upon^^Xhe intended use. 

5. Distribute copies of Handouts^, "Evaluation Techniques and 
Law-Related Education Questions,'' and tell th^ participants to work in 
groups of three to determine the^ most appropriate technic^ue for each 
question. / . , 

33 



• ^. Go over each item, clarifying the correct answer and asking 
participants why tliey selected a particular technique. In some cases, 
more than one technique would be useful. For your convenience possible 
answers are provided below: 

5 ' . 7. 



1. 
2. 
3. 
4. 

5. 
6. 
7. 



I or 4 
I or 2 
1 or 2 

3, 

^1 , 4, or 8 

5 



1, 2, or 8 " ^8. 

1, 2, or 4 " 9. 
Depends on definition 10. 
of effective 11. 
4 or 6 12. 
7 or 8 

Discuss the factors that must be considered in selecting eval- 
uation techniques: cost, number of students (or subjects), amount of 
'time and pe rsonnel the kind of information needed, and the l^vel of 
ob j e c t iv i ty nee ded ♦ 

8. Conclude by distributing and discussing Handout 10, "Major 
Characteristics of Four Information-Gathering Techniques." (If partici- 
pants are not very knowledgeable about' evaluation techniques, you may 
want to use this handout at the beginning of the activity so participants 
can use it to complete the matching exercise.) 



\ 
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Handout 9 . ^ • - 4 o£^ i 

(Activity 6) • ^ " I ^ 

EVALUATION TECHNIQUES AND 
* . LAW-RELATED EDUCATION QUESTIONS 



Worksheet . 

Direct ions : Below is a list of specific evaluation techniques. For 
each of the following questions , identify the technique(s) most appro- 
priate far gathering information to answer ^that question, . 

1 .* Qu>s4;ionna ire/ Survey ^ 5* Anecdotal Records/ of 

2 . Int-erview^^,^^^ ^ Unstructured Observation 

3. School Documerrt^-.,or Records 6, Analysis of Homework or Class- 
A. Structured Observation^ ^ ^. vork 

(a) Checklist """^^^^ 7. Essay 'Test 

(b) Rating Scale ^^^^X^acher-Made or Standarjiized 

(c) Formatted Systems T^t 



1. Do students in your project know the ba^^ f\cts 
from LRE instruction? 



2. Have students' attitudes toward police changed sincet 
the beginning of LRE instruction? i^^) 

3. How interested in LRE are the students at Jackscm 
High School? 

4. How effective has LRE instruction been in tfhe 30 
pilot classrooms? 

5. What aspects of the problem-solvi^lig process^ can your 
students apply to a law-related siliuation? \ 

6. Have the 600 students in the LRE project imj^roved 
their reasoning skills? . 1 

7. Did the teachers in my project, use the LRE materials 
about which they learned at tile awareness work^hop^? 

8. What do your project teachers like and dislike ^out 
the LRE materials they have used? 

9. How and why did the 12 teachers in your project get 
involved in LRE? 

10. Has the level of vandalism at Southern Hills Junior 
High decreased in the past six months? 

11. Do the 300 students in your LRE program have more 
respect for authority and the law as a result of 
that program? 

12. How well have the 20 teachers in your project used 
the LRE strategies that they learned in the inservice 
training sessidns? 
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* 

MAJOR CHARACTERISTICS OF FOUR' ! 
I INFORMATION-GATHERING TECHNIQUES 

* 


1 of 1 






INQUIRY 


OBSERVATION 


ANALYSIS 


TESTING 


Types of 
Instruments 


Questipnnaire 


Anecdotal 
records 


Homework 
assignments 


Teacher-made 
tests 




Interview 


Checklist 


Projects 


e 




J Sociometric 

device * 


Rating scale^ 


Reports & essays 
• 


Standafdized 
tests 




Project ive 
instruments 

- C 


*' Interaction * ' 
records 


Behavioral 
performance 




Kind of Infor- 
mation 
Available 


Opinions 

Self-percep- 
tions 


Performance 

or the end 
• products of 
some perfor- 
mance 


Learning ofut- 
comes during 
the learning 
process (in- 
termediate 
goals) 


Attityde* an4 
achievement 

Terminal 
goraTs 


* 


Subjective 
j udgments 

Affective 

^^^^^^esj)e c ia 1 ly 
a 1 1 i t^iH^K 


Af f ecG.ive 
(especially 
emotional 
reactions) 

o 


Cognitive & 
psychomo- 
tor skills 


Cognitive 
outcomes 

Maximum *> 
performance 




Social 

perceptions 


^^^^^l^tirol^ t e r- 
action pjsy--^ 
chomotor 
skills 


Some affec- 

c 

tive out- 

^^^^-^onies^ 


/ 

V 




• 


Typical be- 
havior 






Objectivity 


Least objective 

Highly subject 
^ to bias & 
error 


^ Subjective 

hii f r An l*j e 
objective ' 
if care is 
taken in the 
construction 
& use of 
instruments 


Objective but 
-not s tab 1 e 
over time 


Most 

ob j et t ive 
& reliable 



Cost 



Inexpensive but 
can be time- 
consuming 



Inexpensive 
but very 

« time-con- 
suming 



Fairly inex- 
pensive 

^^Preparation 
/ time is 

somewhat 

lengthy 

but crucial 



Most 
^ expensive 
but most 
informant ion 
gained .per 
unit of 

. time 



^ From Terry D. TenBrink, Evaluation; A Practical Guide for Teachers 
(New York: McGraw-Hill, 1974),. pp. 136-49. Used by permission. 
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•J ACTIVITY 7: 

/A VARIETY OF EVALUATION TECHNIQUES 

Introduction : Wliile wide variety of evaluation techniques are avail- 
able for use in LRE, each technique hks certain advantages ahd disadvan- 
tages. LRE project personnel need to be aware of thes£ factors so they 
''^can select those techniques that are most appropriate for their partic- 
ular needs* This activity is a detailed follow-up to the^ previous one, 
in which participants selected appropriate techniques and discussed 
general factors that must be considered when deciding on evaluation 
methods. Participants will now read and discuss a handout that explains 
and illustrates various techniques and identifies^ their advantages and 
disadvantages. ~ 

I - 

Objectives ; Participants wilLbe able to — 

1. Explain at least eight different evaluation' techniques th^t 
can be used in LRE projects. -> ^ ' 

2. Illustrate each of these techniques with an example related to 

LRE. 

3. List at least two advantages and two disadvantages related to 
each of these evaluation techniques. 

Time ; 45-60* minutes 

Materials Needed: .-Handout 11 

— ^.^p;^ / . i - 

Procedure : ' ^ " * ^ 

1. Introduce this activity by stating that-^ef ore one selects 
appropriate . evaluation techniques, each techn^^4 must be expliD^red^in 
depth to fully grasp its advantages and disadvantages,^ 

2. Distribute copies of^ Handout 11, "Techniques Useful in Eval- 
uat ing'^Student Achievement." Allow^participants time to read this paper* 

3. Ggnduct a full-group discussion to clarify the main points of 
this paper and to answer any quest ions' part icipants have about each 
technique. 



A. Conclude by' stating that selection of evaluation techniques 
always involves trade-offs — r<^rely can you choose the id^al instrument 
with no disadvantages., The best you can do is make the besjL, possible 
selection and be aware of the limitations related to «that t'echnique or 
select two or more complementary' techniques (if time and resource con- 
straints allow use of multiple techniques). a 
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activity 7) . 

° ' TECHNIQUES USHgUL IN EVALUATING 

PUPIL "XCHIEVEMENT^ "* 

I. INQUIRY TECHNIQUES ' «- . . * 

A. Questionnaires/ Surveys 

Description ; Survey questionnaires are an excellent method for 
assessing the opinions, ""attitudes, and behaviors of large numbers of 
individuals* Questionnaires can be developed'^in two formats — open-ended 
and closed. Open-ended formats present respondents with a series of^ 
questions which they can answer in their own words. This approach has 
the advantage osf not forc;Lng the respondent into.^af response category 
that does not accurately capture his or her .tru(^ perceptions. 'Often, 
^trained scorers can classify "Oj^en-ended responses into categories derived 
from a preliminary scan of all of the responses. 

Unlike, the open-ended questionnaire, closed format questionnaires 
present the respondent with a limited number oi responses to each ques- 
tion. Usually, the respondent checks*, the block •^or circles^ the letter or 
number of the alternative that most accurately describes his or' her 
response. This type of format allows an evaluator to assign each 
response a 'numerical value, and computer cards can be punched directly 
from the^uestionnaires, ,,thus making it less costly than ^the ppen-ended 
instrument. ^ 

A critical factor with both types of jquestionnaires is the selection 
of an appropriate sample. Questionnaires are often used with a sample 
of some group because information cannot be collected from every group 
member. Since a sapple is always smaller than the total group, the 
sample selected must be representative of the entire group if the results 
are to be used to draw generalizations about the entire group. Althougll ^ 
survey questionnaires can be used to gather data fr^om large audiences, 
they ofte'h'do not ^lave the depth that an evalpator migljt desire. ' For 
this reason, interviews are often used to , complement questionnaires Xn 
order to collect' detailed information from a limited" number of a grou^ 
that wa^T surveyed, 

Sample J ; Open-ended survey questionnaire.' 

^ '"^^ V' 

I.* Do you f^el that the police in your neighborhood are fair? 



If you teel that the police are unfair, please explain why you feel 
that way. 



Material in t^his section has been drawn and adapted from several 
sources*. James B. Kracht, "Evaluation," in The Methods Book; Strategies 
for Law- Focused Education , Arlene F. Gallagher, ed. (Chicago: Law*in 
American Society Foundation,- 1978), pp. 126-135; Terry D. Tenbrink,'^ 
Evaluation; A Practical Guide for Teachers (New York: McGraw-Hill, 
1974), pp. U1-U9; and Affective Measures for Educational Evaluation . 
(Tucson, Arizona: Education Innovators Press, 1972). Used by permis- 
sion. 
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Sample 2 ; Closed survey questionnaire. >^ ^ 

,1. Do you feel that the police in your neighborhood are fair? * 
Yes 

^ No e ^ • 

^ Most of the time , 

* ^ Seldom . ^ 



If you feel that the police are not fair all of the time, pfle^ase 

check the res^ponse that best explains why you feel that way, 

■* < 
^ They pick on minorities. 

They^pick oa kids. 

If .you have money, they- don' t "bother you. ^ 

rThey just don't like certain people for no reason. 

Sometimes they ve had a bad day and take it out on everyone^ 

^ Other (specify) ^ 



Treatment of I^esults ; Responses to open-ended questionnaires must, 
be coded before they can be analyzed; this coding can be an expensive 
and lengthy proposition, especially if the 'questionnaire is longer than 
one page. Once coded, the results can be^subject to a variety of statis- 
tical trea^tments, just as the responses to close'd quiestionnaires can be. 

A dvantages ;' "Survey qiiestionnaires ai;e an efficient me.thod for 
collecting data from largje groups. Closed questionnaires can be quickly 
coi\yferted to a form that allows statistical treatment. 

Disadvantages ; Poor sample selection can skew results. Answers* 
are often superficial". Closed questionnaires may force people to make 
choices that do not accurately represent their views. , , 



B. Socioiifetr ic Techniques 

Description ; Sociometric techniques provide a means of determining 
the soc'ial organization within a group. Sociometric instruments can 
provide information about how popular students are within their i>eer 
group. 

Sample : * ; > - 

O 

Write the names of the three classmates you would most like tQ have' 
sit near you, !• ^2. • 3. 

Treatment of* Results; Results can be graphed ^s shdwn below:' 



/ 




6'.}, 
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Advantages : Data can be used, to organize classroom groups and to 
help the teacher facilitate the social adjustment of individual students. 
Data can also be fsed to evaluate^the influence of schpQl practices on 
students' social relations. « & 

Disadva.ntages ; Data do not indicate why a particular social struc- 
ture evolved or what shoyld be done given a certain social structure. 

C. Attitude Scales * 

\ Semantic Differential ; 

I ^ 

Description ; The semantic differential is very effective in measur- 
ing students' attitudes toward a particular concept, gtudents are ask6d 
to rate a concept along a series of .continua formed by pairs of bipolar 
adjectives. ✓'Scales usually cpntain five or seven* positions, with the 
middle 'position representing neutrality. Semantic differential scales 
are easy to fotmulate and are widely applicable. When administered in a 
'pre/past format, semantic differentials provide important diagfiostic 
data, as well as an ^indication of the amount and direction of attitudiji€tl 
chan^ that occurred during a unit. • ^ 

' Sample 1 : 

\ Place an X^in ^he blank of each scale that most nearly describes 
. *you'r reaction to the concept listed below. • ^ 

'~My LRE teacher is: 



1. unpleasant 

2. acWyfe 

3. bad 

A. pleasing 

5. relaxed 

6. detiborate 

7. simple 



1 ' 


i 1 1 




1 1 


1 


1 ! 1 


1 1 


i_J 


L_ 


1 1 1 


1 1 


1 1 


1 


i 1 1 


i 1 


1 1 


1 


I 1 1 


1 1 


1 1 


I 


1 1 1 


1 t 


1 1 


1 


1 '{ 1,. 


1 1 


1 1 



good 



J annoying 

J tense 

J careless 

J complex 



Sample 2 ; 



The purpose of this activity is to measure your feelings about the 
police. There are no right answers. Place a check on each scale at the 
poir^ that b^st represents your feelings. 

^ POLICE ^ 



Valuable ^ 
Bad 

Friendly 
Unpleasant 
Helpful • 
Unfair'^* 



X 



X 



X 



X 



X 



Worchlesb 
Good 

Unfriendly 
Plcasaiu 
Noi Helpful 
Fai'r 



6';T 
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Treatment of Results : Semantic scalea are^scored J)y assigning a 
numerical value of 1 t^the least desirable position and increasing the 
value of each posit^ion by I . In the examples given, negative feeflxngs 
about a value equal I, and positive values ^equaj. 5. A total score can 
be computed for feach concept to indicate overall attitude. In sample 2*, 
the total is computed as , follows: 3 + ,3 + 2 + 2 + 2 + I = 13, Since a 
neutnal score of 3 qn each of the six scales would total 18, it can be 
copcj-uded .that this individual possesses a somewhat negative attitude 
towahrd police. . ♦ ^ ^ 

] Advantages ;- Semantic differentials are adaptable to varying 
research demands and are quick and economical to administer and score • 

Disadvantages : , Respondents can easily fake their responses. Effec- 
tive use therefore requires good rapport with the individuals ^tested and 
a sincere oelief on their part that frank , responses are in their best 
interest. Scales yield ' perceptions of what is, rather than ev idence of 
what'-is . , . ^ " 



Likert Scales : 

'\ Description : Likert scales are al^so effective in measuring student 
attitudes. These scales consist of. a set of statements that students 
are asked to respond to by indicating the extent to which thfey agree or 
disagree. ,The responses can be used to infer 'the attitudes students 
ppssess 'd^oward selected concepts. Like the semantic differential, if 
.used in a -pre/post formatf, the Likert scale can" provide evidence for 
evaluating the effects of instruction. 

"Sample 1: . ' . f 

U.S. Supreme Court just^lces are often influenced by politics in 
.their decivSions. 



Strongly Agree Undecided ^ Disagree ^ Strongly 

Ag,ree , , Disagree 



U*S. Supreme" Court justices should be given a ten-year appointment 
rather than a Lifetime or good-behavibr appointjnent. 



Strongly .Agree^ ' Undecided- , Disagree Strongly 

Agree • ^ * ^ ^ ' Disagree 



Sample 2 : 

Judges are very fair. 
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Sample 3 ; ^tt- [ ^ - 

Circle the response that best describes how you feel. 

r. I can't wait until I am grown up. 

Always Very Often Often Not viry Often Never , 

2, 'I^ f^el good. ' ' ' 

Always Very Often Often. Not Very Often Never 

3. I have a h^rd^time reading math bobks. , — 
Always Very Often Often Not Very Often Never 

A. My parents understand my problems. 

Always Very Of ten Often 'Not Very Often ' Never 

5. I like to be around books.. 

.Always Very Often Often Not VeVy Often Never 

Treatment of Results : Responses can be quantified by assigning the 
number 1 to the least desired response and numbering e'fch response 
consecu'tively, ending with the highest number f or ' the most desired 
response. , 

9^ 

Advantages ; Likert scales are adaptable to varying research demands 
and ^are quick and economical to administer'^and score. 

Disadvantages : Respondents cart easily fake their responses. Effec-- 
tiv,€ u^e therefore, requires good rap'port with the individuals tested and 
a sincere belief on *the'ir pdrtJ that rank responses are in their best 
interest. Sc'aies' yield perceptions of what^^s, rather than evidence of 
"What is . ^ ' ^ * *fe • f 

D. Interviews-' * 

^ — —- .... r 

Desclription ; Interviews can be unstructured or structured. 
Urfstructured' interviews have no established sequence of questions; each 
question flows from the previous one.**" This approach is especially useful 
when general inf oifiyation or preliminary understanding of how participants 
penceive a si.tuatio'p 4-^ sought. For example, an ititerviewer might ask a 
student participating in ,a new program in civil 'law, "How well is the 
program working for you?" The .very general nature of this question « 
allows the respondent a greiart deal of freedom in responding; This flexi- 
bility can help the evaluator identify issues for more^detailed examina- 
tion," 'Sut it makes <*comparison of interviews and analysis difficult. 

* Structured Irtte^view^, or) xhe other handi focus upon Specific issues 
that are predetermined^ by tl\g in t,er Viewer. The interviewer approaches 
each interviewee witk the 'same set of 'questions* "In some instances, the 
interviewed is forced, -ta^select a response from a lipiited ^number gf 
choices presented by the'.iaterviewej: . This is known as a closed format,* 
When?^ the interviewee can -Respond to predetermined questions in his/her 
own terms, the interview is ^.said to * have^an open-ended format . Both 
closed ^nd open-tended structured inter^views provide a great deal of ^ 
detailed information- about a specific issue. However, unless the eval- 
Uator -understands the sl-tuation clearly, the questions may be inappro- 
priate. ^ ' * , 
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Sample I ; Open-encted interview. ' ^ 

Do you -think it's right to' bus ^tudetits to different schocJls 
in oirdfer to\balance ' tlie eprollment racially? ^ Wljy or ,why not? 

2. Would you mind being bussed five mil^s for that ^purpose? Why 
or why not? ' '* ' . - / 

/ . ; . . , • « . 

'3. "for. what purposes would you be. wiping to ride a '^chool bus 
five miles or more each day? - * " - , 

Sample 2: , Closed interview, - 

« » • , / 

• * 

1, Do you think accused criminals should always have the right to 
ball? . • ' * ! 

Yes, always ' ' ? " V 

No, not if they 'are repeat, violent offenders^. 

- N' _2 No, not if they are violent, of fenders ' , 



^ Other 



2, Do you tHink judgers discriminate in^eetting bail for "white- 
collar" crimiTials?^ , • " ' 

• No, they are fair * ^ 



^'Yes, they are easier orv* white-collar criminals 

^es y they are harder on whi^te-collar criminals 
♦ Other 



Advan£ages ; Interviews* provide in-depth a/fective information of a 
highly personal nature about students' opinions, interests, and ajiti- 
tudes. They can be used >f requently' as a normal part^f the communication 
between teacher^a^d student. ^ . * • 

- ' ^ . ^ H ' " " 

N Disadvantages ; Administrattom and scoring can be very t^ipie cotisum- 

ing.. In addition, interviews ar^ ^(^i useful if one ne'eds the same infor- 
mation ab"^out all students, in the class. 



•E. Projecjitive Techniques ' , . 

Description : Projective techniques -Jllow' students, to react to an 
open-ende'd stimulus, ui^th very little limitation imposed by the structure 
fef the instrument.^ Although the results of, projective techniques are 
not . quant jLfiable, they, do provide evidence useful in appraising an - « 
individual ' s feelings*, thoOghts, and attitudes. Three'types of project 
tive techniques wi^l be discussed here: open-end«d titles, open-enHed' 

sentences, and unfinished stories*. * * . . * 

* • J ** 

* Open-ended titles are simple^ and ' effective . §tudents are asked "to 
discuss drally*6r in» writing a response'to a theme. The results, reveal 
•feelings .^an^ attitudes, as well as tlie respondents' basic understanding 
'of the topic.'' . / . . 1 , 
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^ Another simple device is the opeR~e/aed sentence . Students are 
asked to complete a sentence'and, if.^the/ wish, .explain theic position. 
Feelings and attitudes are "especially apparent in these responses. 

'I * * * 

/ 

Unfinished stories are a variatic/n of theT two techniques- desctibed 
above. Usually, a story is read or d film is shown; .the ending of« the 
story or f.ilm is^incomplete. Studer/ts ,are asked to wrUe, act oUt, or 
discusTs possible ei>dings. * Again, /e^lings, altitudes, and values are 
revealed in' students' responses, 

S-ample 3^ ; Qpen-ended tit les 

1. My First Contact with /the Law., * 

2. A Day in the Life of /a Police Officer 

3. . What Should be Done/About Inequality 
^. 4. Changes Needed in Our Country 



Sample 2 : Open-ended ser^ fences. 

1. The local issue that disturbs me most is . . . 
c ^2.^ ^gciice officers areffokay^ except for,. , . 

3. If I did not agree with, a I would . . . 

4. My advice to the Supreme Court would be . '.^ . T . 

5. ' The^ 'trouble with being -honest i^ ; . . 

Sample 3 : Unfinished stories; « 

Tefry had jus*t moved into the neighborhood. Though she miss€;,d her 
.old friends, she was really beginning to like the new plaEe. Her 
fa*ther had a better job, their house- was a, lot nicer, school was 
going Well, and she had made 'some new friends^ In fact, George and 
Judy had become great friends. They lived next ^4bor to each other 
."^aBout eight blocks 'away from Terry. Terry had been' asking her 
mother for the p)ast two weeks if she could go home, with them after 
school to play basketball. The answer had always Ibeen' no , untij 
today. .' ' y ' ' . • 

As Terhy walke'4 home with George and Judy, they explained that they 
usually stopped by the drugstore to ''pick upr'* a few candy bars. It 
was jTist a "little game" they played, they said. Terry told them 
she had enough 'moh^ to buy candy -ior a!^l of t'hem. Judy ^aid the 
idea was to see^how much you could "lift" without, getting caught 
. • and t^Ia^t Terry was-, "chicken", if she'didn't go along, with the pla'n,. 

t 

1. What do yo^u think'Terry did? * ^ " ' ' " \ 

2. Why -do you think she did t'hat? * 

3. ^ What do yoo think you would do? 

Treatm.ent of' Results ; Results ^re* often difficult to interpret. 
One way pf Creating Jthe results is to fexamine responses and classify 
thejui. according- to level of. ego development. Often, a clinical psychol- 
ogist ' s* assistance- will*' be needed in.* -interpreting the results obtained 
from' projective techniques, t *.w . " . • • 

Advantages : Faking responses is difficult. ^Techniques provide 
information aboxit students' feelings, biases, and reasoning%)a.t terns. 
' . ' ' . 
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Disadvantages ; Information obtained is often unreliable, and 

results are difficult to interpret. Usually, a clinical- psychologist 
shduicj interpret, the data. * , - 



LI, OBSERVATION TECHNIQUES 



Teachers and students can use a variet)^ of observation techniques 
in their classrooms. These* techniques can be cla§sified into two major' 
groups: structured and unstructured. Structured observations, require 
use of instrum^nts-^-checHlists, rating s'cales, or some other Structured 
format.' Unstruc^tured ^observations do not require use of instruments, 
although some recording devices may be developed. The *discussion beloy 
focuses on unstructured observation, use of aftiecdotal records with 
unstructured observation, and various methods of structured observation. 

<»A> Unstructured Observation 

• . f ■ z ' 

» * ' ^ ♦ 

Description : Mos't teachers use unstructured observation every day, 
although they may,do. sp unsystematic^ly and with less-than-op timal 
efficiency". At the' end of the day, most teachers can comment, "Gee, 

* thaf^ lesson^ was'- supe^. The kids really responded" or "What a dudi 
Today's lesson. started out badly and went downhill from there." These 

^ types of comments are based upon teachers' observations of the behaviors 
students "exhibit in their^classrooms. 

By having colleagues observe and systematically record what they 
see happening in classes, teachers can learn a great deal about the . 
general climate and operation of their classes; for example, the general 
level of pa-rticipatory skills, content knowledge, level of interest, and 
group work skills. Unstructured observation can also provide detailed 
insights into teachers' majnagemect and instructional styles and students' 
resfons^es to. them. ' ^ 

The most* general unstructured observation technique seeks to record 
.as accurately as possible the comments and behaviors of students and 
teachers during a lesson.- Descriptive statements such as "three students 
^re looking out the window and four are playiilg cards" are preferable to 
impressionistic statements ("some istudents seem bored") . - . 

Sample : ' ' . 

Teacher writes on board: USMC vs. Robert Garwood 
« 

' Teacher: Good morning. Tcday we're going to consider a case in 
' .\ corrective justice, but first let's review. As you 

know, weVe been studying the concept- of justice, what's 
fair. What specifically is corrective justice? 

Student It: A fair response to a wrong or injury. 

Teacher: Okay. Can someone -give me an example? 

• Student B: A jail term for armed robbery. 



'<0 
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Treatment of Results ; Careful analysis of unstructured observation 
notes will oft^n suggest areas that^ may deserve closer attention and 
additional observation. For example, a teacher may notice that he/she 
mediated every student comment, never giving students, an opportunity to 
interact directly with one another. Unstructured observations might 
also suggest that particular students work poorly together. Additional 
observations might demonstrate that placing these .students in other 
groups improves their group work skills. Instructional objectives should 
be kept in mind during transcript analysis. 

Advantages ; Unstructured observations can 'pravide a great deal of 
injcormation of import to classroom teachers. The flexibility of unstruc- 
tured observations allows collection of information about a wide range* 
of topics, and no time is required for instrument, development. 

< Disadvantages ; Unstructured observations require the' cooperation 
of another adult ^who, if untrained, may have difficulty reccJtding the 
great number of staten^ents and events that occur during a class. In 
additioni; analysis of transcripts takes time. 



B. Anecdotal Records 



Description ; Anecdotal records present bri^f sketches of the . 
actions 6f individual students, small groups'' of students, or students 
confronted with a situation of very Ijimited duration. While maintaining 
such records is time consuming, they can be a valuab/e aid iri appraising 
student growth. A folder containing a series of sneets', each headed 
with a student's name, provides- easy access to the data. Entries can 
then be made during or at the end o£ the school day. The record shouljd 
include the date of the observation and a brief, objective description 
of the behavior obse^rved. L 

• • ; ' I' 

Sample' I : ^ • - • ' 



|er!c\ 



Name 


Llovd Larkins 


'Dates. N'0\.2toNov 26th 


? 


\ 

Date 


Observation 


' Commenc 


.11/02 

/ 


. Did not pay attention and • 
created discipHne probknn 
.during intfoduaion to Juvenile 
Law Unit. 


He seems as bored with this as 
with everything else. 


11/08 


Still has not participated in clasb 
» discussion. ^ 


• 


11 'la 


Participated m discussion of ca:>c 
study on juvenile ot'fendcrs. 


Intcresc sccm:> co be picking up 


11/26 


Volunteered to take part in role- 
play of*juvenile hearing. 

i . • 71 


One of the few times this year " 
that I've had willing coopciation 
from him. 

♦ * * 
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Sample 2 : 



Class : , Fifth Grade 
Date: IflZl^l 



Pupil: 
Place: 



10 of 20 



John Smith 
Classroom 



Incident: As class started, John asked if he could read 
a story he had, written the day'1)efore. I gave him permission 
and lie c^me to the front of the room. John began to read the 
story in a very low voice. When*he had been reading for about 
30 seconds, Sharon, who was sitting at the back of the room, 



asked John to read louder. 



At this. point, John tore up the 



pages he was reading and went back to his seat. 

Interpretation: John appears to enjoy not only writing, 
but sharing his creations with other people. It seems, how- 
ever, that he is very nervous and gets upset easily when he 
receives the slightest criticism, as noted "by his tearing up 
of his story and refusal' to continue. 



Treatment of Results : Analyzing results can be 



difficult. 



The observer's interpretation ' af .observations can be no£ed wajt? each 
anecdotal record, ^as shown in the two samples above. Analysis%f change 
over,, time can be done afteV a number of anecdotal records have been 
collected. 



Advantages : Anecdotal records provide a description of actual 
behavior in natural settings. They can be used to collect inforjpation 
on very young pupils or others who have limited cpftununication skills. 

/ Disadv.antages : Time required to maintain an adequate system of 
records may be substantial. Maintaining objectivity in observing and 
recording pupil behavior is difficult, as is analysis of the data. Care 
must be taken to obtain an adequate sample ol behavior. 



Structured Observation 



By def initjion , unstructured observations are somewhat diffuse and 
unfocused. Structured observations, on the other hand, rely upon a .range 
of observation techni^iues that focus on specific topics. For example, 
in order to gather information about ,the types of questions students ask 
police officers,* a teacher might construct .a checklist of such categories 
as the following: points of law, career information, and attitudes 
toward youth. Once criteria were established fo.r what constitut'&s an 
appropriate response for eacj) category, the teacher. or a colleague could 
observe 'the class the »next time, a police officer was present, placing a 
ch^ck in the correct column for each question posed by students/, ^^^e 
checklist would provide information only about studei^ts' quest ions i^oJ> 
about their answers" to questions, asked l?y thfe- officer or anything else. 

The following discussion focuses on three ty])es of structured^ 
observation techniques; checklists*> rating scales, and. Structured 
formats. , ' * ** 
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Description ; Checklists can be developed for us^ by the. teacher or 
by students. Checklists are flexible, since they are based directly on 
the performance objectives of the unit being taught." They also provide 
objectivity and a semipermanent record of student achievement. Check- 
lists for use by students can be developed cooperatively, resulting in 
students* having to focus on the unit objectives and assume , some^degree 
of responsibility in assessing their own learning. 

Sample 1 : Self-respect checklist (to be completed by teacher). 

Student's Name: John Smith 

Check ( ) if observed. 



1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 
10. 
11. 
12. 



Sets goals for self (without teacher solicitations)^. 

Seeks help when needed (from teacher, aide, or other student). 

Assumes responsibilities at^: school. 

Fulfills accepted responsibilities on time. 

Corrects his/her own work reliably. 

Uses time wisely. 

Keeps self well-groomed. 

Is courteous. 

Is eager to lead the group in discovering new knowledge. 
Responds positively to constructive criticism and suggestions. 
Cleans his/her desk and flooi: area on own initiative. 
Exhibits positive social behavior (leader^or participator). 



Sample 2 ; Checklist for police unit (to be completed by teacher), 
Names of Students 



< e 

< I a 



BthavioT'to be obstrved 



cj : 6 



1 Bfinp in newspaper article 
relaced to poitce. 



2. Seteccs one of the books about 
police from library or reading 
table. ' ^ 



3 Willingly volunteers far one of 
five optional work groups pre 
panny room display on police . 



4. Fulfills /esponsibility in work 



I. I 



5 Worici cooperatively with othcn. 



6 Asia questions of policeman 
and ocher juesti durinj iheir 
visit ^ 



7 Keeps notebook throughout 
unit 



5 \ctjvflv paitjcipatcs in cUi* 



•Check each child three times durmr this unit 

7J 
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Student checklist. 
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Name Penny Ponder 



Unit Police 



1 — Oucst^nding 
h 2 Very Good 



3 — Satisfactory 

4 — Needs Improvement 



Week 



I parucipatcd in class discussions. 



2 



I aslfced questions of Officer McKinney and 
oth^^r guest speakers, 



I did not interrupt^ others when chey were 
talking, 



n — 

enn boarc 



I brought news clippings for the bullet' 



I volunteered for a work group. 



1). 



I did my share of work in my group. 



7 I read a library book about police work. 



8. I kept my notebook up-to-date. 



Treatment of Results , Using a checklist allows a teacher to quickly 
calculate the number of times students exhibit the behaviors of interest. 
Totals for each student can be computed and the results compared at vari- 
ous points during t-he school year. These comparisons will reveal any 
changes over time. 
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^Advantages: Checklists are useful in evaluating those skills that 
can be. divided info a series of clearly defined, specific actions^ ' They 
can serve as a convenient method of recording evidence of growth toward 
specific learning outcomes. 

Disadvantages ; Use of checklists is susceptible to respondent bias. 
Checklists do not indicate the degree of behavior, simply^ its presence 
or* absence. Checklists are not useful in summarizing a teacher's general 
impressions concerning the personality and adjustment of pupils. They • 
can also be cumbersome to, use with a' full class of Students. 



Rating Scales : ^ ^ 

Rating scales are' another type of structured observation instrument, 
Ihese scales focus upon specific issues and require assigning a score to 
students* behavior or comments. For example, a teacher who wants to 
assign letter grades to students 6n the basis of the quality of their 
comments in class discussions might devise the following rating scale: 
asks thoughtful/thought-provoking questions , asks factual questions, 
asks questions unrelated to the topic. Scale points can be described in 
varying detail and need not necessarily be the same foir all questions. 
They may .change depending %pon the nature of informition sought. The 
example below indicates a different scale than the oiYe-<iescribed above. 

Sample: 

Student: Jo/in Sm///i 
Scale Points: 1 — Excellent 
2 — Good 
• ' 3 — Average 

4 — Fair - 

5 — Poor ^ - 

1« How would you rale the student's ability to get along with 
his peers? , ^ . ' 

1.2 3 4 5' , 



\ 



2* Koil^ would you rate the student's creative potential? 
1 2 3 4 5 

3. How would you rate the student's attitude toward the genr- 
oral school environment? 

1 2 3 4 6 



Treatment of Results . Numerical values can be assigned to each 
point on a rating scale, allowing averaging of class scores. A compari- 
son of these averages as well as individual scores at different times 
permits rapid analysis of class and individual change. 



7J . 
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Advantag^es r Rating scales provide a common 'frame of reference for 
comparing all .students on the same set of characteristics. They also 
provide a convenient .method for recording observers' judgments'. 

Disadvantages ; Many observers exhibit a tendency to rate all indi- 
viduals at approximately the same position on the scale. In addition, 
the rater's general impression of a person njay influence how he/she rates 
that person on individual characteristics. Characteristics may be- rated 
as more or less alike than they are because of the rater's belief, con- 
cerning their relationship. < i 



Formatted Observation Sy:stems ; 

Formatted observation systems are tightly* structured observation 
methods that attempt to measure the j^resence of certain be'haviors. For 
example, a teacher interested in the degree to whicfh he/she uses certain 
questioning strategies might construct an observational farniat with the 
following key: factual^- F; inferential - 1; synthesi's*^- S; analysis - 
A; evaluative - E. The observer would record the appropriate^-* let ter 
_ each time the teacher asked a question which fit one of these categories. 

Sample : Interaction analysis observation system. ' ' ' 



TEACHER TAU , 
Blxecc Influenct 

'1. ^ ACCEPTS FEELING: accepts and ci^rlfles ch« feeling cone o£ 
Che scudencs In a nonchreacenlng manner. Feelings ^may be 
poslcive or negative. 1 

It 

2, PRAISES OR ENCOURAGES:, praises or encourages scudenc acclon 
or behavior. 

3. ACCEPTS OR^USES IDEAS Of STUDENT: clarifies, builds on, or 
develops Ideas suggesced by* a sCudenC. 

A. ASKS QUESTIONS: asks questions about concenc or procedure 
with Che Incenc chac a scudent answer. 

Indirect Influence 

5. LECTURES: ' gives faces or opinions abou( concenc or procedure. 

** . ' 

6. GIVES DIIIECTIOHS: gives dlre<:clons, cocmunds, or orders wlch 
which scudenc is expected cq comply. 

7. CRITICIZES OR JUSTIFIES AUTHORITY: makes scacemencs incended 
Co change scudenc behavior or explain why che \eacher Is ceklog 
a parcicular action. ' ^ 

'student tALK ' ' 

^. RESPONSES: makes a prediccable response co che ceacher. 

9." INITIATION: " Inlclaces Ulk, incroduces new ideas, or makas 
unpredictable responva Co ceecher. 

4 

^10. SILENCE OR CONFUSION: pauses, periods of silence, and periods 
of confusion in which comsunicetlon cannot be underscood by 
obsar^r. 

No scale Is implied by che nuabert. They^ara merely classlflcacory, 
deslgnaclnt a perclcular kind of communication. * ^ 
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Treatment b.f Results ; After a formatted observation has been com- 
pleted, the total number of behaviors <an be calculated for each dimen- 
s-ibn"" measured by the form. Repeated use of the same form allows compari- 
son of ijesults, which will reveal any changes in student 'or* teacher 
behaviors. " ^ ^ 

Advantages ; Format tfed observation systems provide a systematic 
method of observing and recording behavior. A large sampling of behavior 
Is possible, and the data obtained c^n be analyzed statistically. These 
systems are adaptable to the situation being observed. 

Disadvantages ; Data gathered through this method may be unreliable 
unless the observers are highly trained. The observer ' s^ bias may be 
reflected in recording behavior. A large amount of data is required, 
and the procedure may be' difficult to use wl^th large 'giroups. 



111. ANALYSIS TECHNIQUES 
A. Student Work 



Description ; Analysis of student work provides another technique, 
for assessing student progress in a variety of dimensions. Homework, 
classworky projects, and essays can all provide valuable data for analy- 
sis. 

Sample^ 1 ; Homework assignment. 

Read the following case^ study and write answers to these questions: 
1. What decision was made in this case? 

2% Who supported and who opposed the decision? 

*\ I, 

3. What did people do in trying to get others*^ support? 

'I . . 

Sample 2 : Project assigni^ent. 

In a^'group of four students, plan and enact a situation in which a 
.cement mill has- been forced to lay off 10 percent of its workers 
because of environmental hazards. The roles depcited will include 
the company president, a union leader, a worker, and an^ environ- 
mental protection advocate from tlte, community, x 

Treatment of Results : In using these sources of information, the 
analyst must establish consistent criteria to be applied^ For example, 
if students are writing an .essay about,,, how they — as police officers — 
would handle an anonymous tip about a crime,' the grader should h^ve clear 
criteria about the regulations governing search and seizure. .When eval- 
uating any ^student work, criteria should be used; if grades are given, 
the. criteria necessary to achieve each grade should be clearly stated. 
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Advantages ; Analysis of student^work can provide precise informa- 
tion concerning what students know or do not know, can do or cannot do, 
a^ they are learning. It permits teachers to evaluate tasks requiring 
more time than a normal class period allows. Evaluating student projects 
can provide information concerning cognitive, nonverbal, and* social > 
•learning for an individual or^ group of students. Some variables can be 
controlled for in-class projects. 

Disadvantages : Analyzing and interpreting the work of every student 
in the class is time consuming. Many variables cannot be controlled, 
and determining how or why learning occurred is difficult. 

B. School Documents and Records 

Description : School documents and records can provide yet another 
valuable source of information about students. For example, law-related 
education program personnel may be interested in the impact of the pro- 
gram upon the citizenship behaviors of participating students. School 
records could provide information about referrals to the office^ 
unexcused absences, and tardies which might be indicators of civic behav- 
ior. 

Treatment of Results . Some records can easily be converted to 
numerical data while others, such as written comments by forner teachers,, 
may require qualitative analysis. 

Advantages : * School records are both s^tematic and longitudinal. 

Disadvantages : Privacy considerations may restrict access to 
records. Individual maturation and other variables may account for 
changes from one year to the next. 




IV. TESTIIiG 



A. Teachat^-made Tests 

Description ; The teacher-made test is probably the evaluation 
device most commonly used in appraising the success of law-focused 
education programs, both at the classroom and the district levels. 
While most teachers construct and .administer their own tests, few have 
had the benefit of any training in this a^rea. This ^section contains 
some general principles regarding the construction of teacher-made , tests , 
with -specific attention to simple recall or completion, alternative- 
response, multiple-choice, matching, and -essay items. ' 

The well-constructed test possesses the basic characteristics of 
validity and reliability. Validitjf refers to *the extent to which a test 
.covers Che material that was taught and the. extent to which it measures 
the actual achievement of performance objectives. Thus, if the fi^form- 
ahce objectives for a, uni\t specified the use of complex th4nking abili- 
ties, a test that only' measured recall would not be valid. In short, ^ 
valid test ^measures what it is supposed to measure. Reliability refers 
to the extent to which a' test provides consistent results ""in successive 
uses. 
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l:^e«-t---shQu^ld_also be written at the reading and complexity level^ 
appropriate to thepaTnrarla^^--grQU beting tested. If the reading' level 
is too high, the resulting poor scqres^~aT6~ndtie^^tx>_rea^ng difficulty 
rather than> lack of knowledge of the content. -If a teacKeT^irs— unsure_^ 
about the-appropriateness of the readabilfty level, it is best to read^ 
the test aloud to the students. 

All directions on a ^est should be brief and simply worded; All 
questions should be written in a similar fashion, avoiding, direct quotes 
from a textbook or misleading questions. The former encourages roCe 
memorization and the latter encourages students to guess. While these 
general principles ^re" important to consider in constructing all tests, 
a number of specific guidelines should be kept in mind for specia-1 types 
of items. • * . «^ 

Simple recall items measure a pupil's ability to remember important 
names, events, dates, or concept!. Such items are usually wtitten as, 
short questions followed by blanks for the students' responses. Care 
should be taken to ensure that there is only one correct answer. 

Sample 1 : > ' ^ 

Who was the Chief Justice of the U.S. Supreme Court at the time the 
Marbury vs. Madison case was heard? - ^ 

Completion items are similar lo simple* recall items in that they 
are questions followed by blanks fox the students to fill in. To avoid 
giving clues tct the answer, blanks of uniform size should be used and 
use 'of "a'* or "an" before blanks should be avoided. No more than two' 
blanks should be left in any statement. * * ^ 

Sample 2 ; 

An appeal to the ?ourt to see if a person is be^g illegally held 
in jail is called ; . " 

» *^ - « ' » 

AiteTnative response items can be written in a variety of forms, 
such as trueVfalse, yes/no, or right/wrong. Thi^ type of item should 'be 
used when there i& a choice of only two responses. Specific determiners 
such as always., never, all, and none shouid'' be avoided. Most students 
can easily infer that such items are false. Having' ^pupils write a T or 
an F for an Answer should also b'e avoided; it is surprising'^how similar 
the two can be made to look. Scor.ing can be made easier by placing all 
blanks either at the left' or the right side of the page. Alternative 
response items are especially useful for, evaluating' learning at lower 
levels cf the cognitive domain, as well as in appraising stereotypes and 
misconceptions. • ' ' 



Sample 3 : '>^^. 

The 'U.S. Suprqme Court has' held that juveniles have a right 

it to trial by-^ry . ' ' • 

Multiple '■-^oj.c e i t ems coi\^st ajf two parts:' m incomplete or stem 
statement and seVeral possible answers. Of the possible responses, one 
is the correct, or the best-, ■response, and the others are distractors. 
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^ Wh'en constructing ^^multiple-choice^iteros^^ the major portion.of> the 
statement should be placed in the stem rather than in the answer 'choices. 
Possible answers should be' plausible and 'of approximately^ the same 
length. Absurd responses and exceptionally long or short. responses will 
tend to help students easily eliminate some cf the choices, v. . 

-.-^J1ul'tlple--choice items are probably the most widely used of all test 
items. '"ATpT^j^^ t^jje^are^ime-consuining -to construct, they 'a,re quickly, 
easily, and objectively szofedir — Xhey_can be written to^measure various^ 
levels of learning .in the cognitive domairT^^ndl-are^appropriate for use " 
irt elementiary-^ schools as well as in secionciary schodls~.^^ ^ J 

Sample 4 ; ' ^ * . 

The significance of the Brown v. Boarii <5f Education decision was 
that it ^ ' 

A.< liberated schools ftom local control: 

B. struck down the. "separate, but equal" doctrine. 

called for immediate busing to achieve racfal balance. 

brought a -quick end to all types of segregation. ^ , 



Matcbi-ng items can be used to evaluate . students ' abilities to 
associate .such items as'^terms and definitions and princ^iples and de/ini- ' / 
tions. A teacher^^an provide clear directions that include 'the basis on/*' 
which tljie matching is to take place.' In Writing matching items, items 
in each column should be hothogeneous. There should be more responses in 
the second co}*umn than in' the first column, jthus ,preventi;ig pupils from 
-using a process of eliminati^on to* find the correct answers. 

Sample 5 ; * / " - ' ' * ^ ^ ^ \\ 

Directions:. Match the following items. ' " . * 

' Provides foj: right of women to vote 1. 'piirst amendment 

Provtdes for freedom of* ^peech^ presis 2. Second amendment 

' N Provi;*s for protection aga^t^st self- 3. Foui^-th amendment 

incriminal:lon *' ' , J ^' Fifth amendment 

> Provides for right of eigl^teen-year-olds v5* ^^ixth amendment 

/ * to.. vote ' / . , ' . 6a Eighth amendment 

Provides protection against "cruel and ^.^^.^ 7. Ninefeenth amendment 

unusual punishments , ■ ^ * , ' ^ 8. Twentieth) amendment 

• ^ Provides freedom ^ro.m unreasonable 9. Twent>r«^f if th amendment 

search and seizure ^ ' 10. Twenty-sixth amendment 

* Essay items are very useful for , measuring analytical, synthesizing,, u^. ^ 
and evaluative skills. They are alsa' useful in appraising students* » 
ability to organize .and summarize information and to apply concepts in, a ^ 
new situation. They are a],so useful in measuring , attitudes toward con-"^ 
troversial issues and .ideas. The advantages of the essay item include 
providing more freedom for students to express their individuality and - 

5, imagination."^^ In addition, -essay questions are more easily and quickly 
prepared th§n* are others. Disadvantages include the time and difficulty 
involved in scoring Hhe answer, ^s well as the limited coverage of the 
content. C'are%should be taken that students do not exjjeriente undue 

V difficulty|^wi_th: writing and grammar. ' ' j» ^ 
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Pi teacher can overcome some of Cl?e 'drawbacks of essay questions by 
restricting 4:heir us^ to^^learning outcomes that cannot be readily mea- 
sured through ^other typfes , of items. , Each' question should^ clearly 
ide^^^fy the students' task ,and should indicate the approximate expected 
length and, the time allowed to complete*the *answer. 

\ * • ' ^ 

Sample 6 ; c | 

•Write one or two paragraphs describing' the similarities and dif- 
ferences between the vigilant^ trial of Henry Plummer and a legal 
court trial. Focus^ especially on judicial procedures and rights <ff 

». the accused.' • " 

*» * 

.Treatment of Results : le.acher-made tests are usually graded and 
assigned a numerical score, often a percentage of correct ' responses. A 
review of these scores will indicate how welj. the class as a group per- 
formed -on ^the test. It will also provide^ an indication of how well 
incfividual^students achieved the objectives measured by the test. - r 

, ' Advantages *^-^t^ac he r-made tests can be constructed to address the 
specif iop object ive|\,df ^-program and can be adjusted to they level of 
students. ^ , ''^ r \ I ' ^ 

^, ' • 

Disadvantages : Teacher-made tests are often poorly constructed. 
Fre<fu§ntly there is no opportunity to check for re^liability or validity. 
' , ^ ' ' . * 

B. Standardized Tests * . ' 

Dd'sc rj.pt ion : A wide array ^cpmmercially. available tests have" ' 
been developed. Some of these have been developed in conjunction with 
specific curricular materials, while others* attempt to measure achieve- 
ment generally. National, state, and local LRE projects often^have tests 
related to their specific projeiEt objectives. \Some of these tests have ^ 
been subjected to rigorous- statistical analysis, while others have not: , 
If -a teacher can locate a> reliable and valid test thatjaeasifres the goa^s 
of his/her program, -using that test is mu'clf easTer i^han\developing a^ 
good one. Unfortunately >- teacliers are often unabl^^ch-44rnd a test tha:t \ 
addresses the s*ime_o6'jectives as their programs. Selecting appropriate 
items from several instruments may be possible, however.. When develop- 
ing or selecting a'^ test, it is critically important that each it^m 
included be specifically tied to an objective; if thi's is not tr^e, 
results may be poor^.. / 

m / 
■ Treatment of Results ; ^ Numerical scores are almost always assigned 
for standardized ^tests. These scores can be compiled to produce a class 
profile. In addition, standardized tiC^sts t:an provide informa't^on about 
how well individuals and classes did on the' test compared -t^i other 
students throughout the country. ' / 
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Advantages^ . Administration and scoring instructions aire 'usually 
provided with .standardized tests. The test is clearly defihed and can 
measure cognitive achievement, ' aptitude, interest^- and a variety of 
personality traits. information from standardized tests ha^ known 
parameters^, such as average performance for v^^rious groups and validity 
^and reUability estimates. Standardized tests can be used to pompare» 
one's students with others in* the nation. The resulting data can be 
used .to assess program impact. • 

Disadvantages : Standardized. tests may ^address ob^ ectives 'that are 
different from the : individual teacher's. Locating appropriate tests^'may 
be difficult, and the costs may be unfeasible. Administering, scoring*,, 
and- interpreting the results of standardized tests are somfetjnves compdi- 
* cated. , ' ' ' * ' 



ACTIVlir 8: 
.INDICATORS AND TECHNIQUES ' 



Introdfuct ion : Participants are no*w ready to apply the ideas and informa- 
tlon 1-earned in the previou-s two ^<;tivities to the^r own LRE- projects. 
The second page df the "LRE Project Management and Evaluation Form" Is 
used to help par^cipants determine, indp-datdrs rela'teS tq their project 
objectives and to select appropriate tig^^iques for ga^thering data' on 
those -indicators.* . * . i 



Objectives :. Pdr,t»Lcipants will be a^ble to — ^ • . ^ 

1. Identify the indicators of achiecvement related to each of their' 
proj ect objectives. 

f 

2. Select ^appropi;ia^e evaluation techniques for gathering data on" 
each indicator. \ ^ \ ' 



Time: 2-3 hours 



Materials Neede^ c Handouts 2 And 12, newsprint and jpiarking pens 

' , ^ ; ^ ^ / 

Procedure : ^ f * * 

•I. Introduce this activity by distributing copies of Handout 12, 
,'%RE Project Management and Evaluation Form: Page 2 Illlis tration. " 
Thii handout providfes an .example af a completed form on which indicators 
and techniques are specified and related to .each other . State that this'^ 
is an exampple of what participants will produce from their project work. 
Ask participants to state whfether they feel that the particular tech- ^ 
niques selected on. this sample form are the most appropriate for. the 
given indicator s^ '•Participan»ts may have some minpr criticisms, but 

generally these a^e appropriate te.qhniques. ' \ 

» * . * • ' 

- 2,. Tell participants to complete their blanK copies of .pa^e 2 of 

the "LRE Project Management and Eyaluatit>n •Foml* ' (distributed in 

Activity 2) for their projects. State .that <they- wili* also, need to refer 

to their completed forms on objectives anci activities, (page 1 of this 

f orm) . ( Suggest that-.they also^ reread the* evaluatipn strategies and 

instrument sections of^ the" evaluation model; ^ ' • ' ^ 



3. Circulate and provide help as needed during this work session. 
As gr9ups complete a section of this form, check their work to^make sure 
they are making the most " appropriate selections aad are .writing them 
clearly. 

A. Conclude this session one of two ways. If the group- is small, 
ask each project to present their complected forms (either duplicating 
copies or placing -them on poster paper) to the full group for comments. 
If the group is large, ask par^ticipants to identify problems that arose 
during the work session 'and important points to keep in mitid when com- 
pleting this form.. ^ 
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LRE PROJECT 
MANAGEMENT AND EVALUATION 
PAGE 2 ILLUSTEIATION 

' i 



FORM: 



^ • INDICATORS. 



EVALUATION TECHNIQUES 



l.ll 



1.12 



2.-11 



2.12 
-3.11 
3. 12 



Teacher^' knowledge about the existence and' 
nature of the Street Uaw ,. Constitutional RightS : 
andl Law in a Free Society materials and ♦ 
activities, ^ 

J 

Teacher perceptions of the usefulness of the 
workshop. 



Teachet behavioral performance in using the 
case studies, role^plays, media,' mock Ujrials, 
community involvement, and discussion * 
strategies in thp LRE materials. 

Teacher perceptions of the usefulness of the 
workshop, 



Student knowledge about the functions of law, 
criminal justice, civil law, and constitutional 
rights.' ^ 

Student attitudes toward the police, the courts, 
constitutional ' rights, and the rf^ed ^r law. 



1.111 Short-anawer completion test on a 
pret^st/posttest basis^ 



1.21 Workshop evaluation form (ratings 
scales |j|nd open-ended iteith»5 on 
i posttest basis. 

2.111 Observation analysis form used ' 
during practice sessions and class- 
room teaching. 



2,12J Workshop evaluation form (rating' 
ecales and open-tended items) on a 
posttest basis. 

0 

3.11b Multiple-choice *test ori a pretest/ 
posttest basis . 



3.121 Likert-ty^e attitude survey 
^ pretest/'posttest. basis. 



on a 
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ACTIVITY 9: 
USING LIKfiRT SCALES TO EVALUATE j 
^STUDENT ATTITUDES IN LRE- ' • 



Introduction ; .Likert scales and surveys are one very useful and effi- 
cient way of m^suring attitudfe change in students or teachers. While 
this technique is frequently used in education, including LRE>, it is not 
commonly used correctly . Developing a good Likart attitude survey 
involves ^careful application of several principles. .This series of 
activities is designed to teach participants those principles and .skills 
so they can develop gijodr surveys for use with their LRE projects..^ The 
skills taught can also be applied to 'evaluating existing attitude sur- 
•yeys and modifying them to fit^the needs of par ticular ^^roj ects . 

Objective s; Participants will be able to — 

— — " ' — - \ «a _ ' 

1. Identify the specific attitude scales underlying a set of 
J^ikert items. . y ^ 

' ^ 2, Eval^uate"^ a set of individual Likert items \v applying a spe- 

cific set of criteria. * 

3. Icf^ntify appropriate and inappropriate uses of Likert surveys. 

A. Describe and illustrate several different types ♦of response 
formats for use Likert; sur'veys. 

5. Develop a Likert scale for LRE attitudes and write a set of 
Llkiert items for it. 

Time: 3 hours 



Materials Keeded : Handouts 13-17, newsprint and marking pens or chalk- 
boara ^nd chalk ^ ^ • * ' ■ 



Procedyr^e ; ^ 

1. Point out that this session will not « involve (a) writing 500 
bcliaviorally ^ stated affective gbjectives, (b) learning any * complex 
statistical procedures, (c) discussing the theoretical aspects of atti- 
tude measurement, and (d) using various methods of^ measuring Attitudes . 

Stress that thes-of are all worthy activities, but that this session will 
« 
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focus on more manageable objectives: (a) learning to use one very use- 
ful and reliable (though not perfect) method of measuring student 
attitudes—Likert scale$ and (b) working individually and in small 
groups to analyze, evaluate, and develop Likert scales. 

2. Provide working : definitions for the . following terms^: 

(a) attitude—a set of beliefs or feelings focused on a specific object 

or situation; (b) Likert item—an example is provided rather than a 

precise- definition: **This has been a very worthwhile conference" 

Strongly Disagree Neutral Agre^ Strongly 

Disagree Agree 

(c) Likert scale—A collection of Likert items designed to measure a 

specific attitude; (d) Likert-type attitude survery—A collection of 

several Likert scales ^designeci to measure various specific attitydes 

toward a common subject. Distribute Handout 13, '^'Juvenile Justice 

Classroom Poll," stating that it is an example of a Likert-type attitude 

survey developed as an instructional tool rather than an evaluation 

instrument. • . 

3. State that good attitude surveys are not comprised merely. , of 
AO separate items, but of carefully constructed sets of items designed 
to measure -well-defined scales. Distribute copies of Handout lA, 
"Political* Attitude Scale," which includes sections of a political atti- 
tude survey developed for 'high .school social studies curriculum. Ask 
each participant to match the^ 16 Likert items with the four scales they 
were designed to ji^sure (Sense of Political Efficacy, Political Trust, 
Egalitarianism, and Governmental -Support) . Go over the answers and 
discuss problems encountered uin doing this exercise. For your conven- 
ience, the answers are provided below: 





C 




9. 


C 


2. 


D 




10. 


A 


3. 


B- 




. 11. 


B 


A. 


A , 


< 


12. 


C 


5. 


A 




13. 


A 


6. 


C 




M. 


B 


7. 


D 




15. 


D 


8. 


D 




16. 


B 



A. Tell' the participants to divide into groups of three. Distrib- 
ute cdpie$. of Handout 15, "LRE Attitude Survey," which contains 16 Likert 

* items drawn from several J.RE surveys. No definitions of scales compris- 
ing the survey are providejl. Tell the groups infer 'the four scales 
which those 16 *lt'ems; seem to measure. Go' over the answers with' the 
whole group. Provide the scales which the developer of "the instrument 
intended^ to measure with those items., (provided below) and briefly sum- 

c marize the main point to be derived from these last two activities. 

1. ^Support" f or constitutional rights 

2. Respect for law ' ♦ , 

3. .Support for constitutional rights 

4. Attitudes toward police 

5. Respect for law ^ . . , 

6. Attitudes toward police - . 

f> . ' 

7. Support for. constitutional rights * • « 

8. Respect for ]aw and support f'or constitutional rights 

9. \Attitudes toward po'ii^ie and support for "constitutional rights 

10. Attitudes' toward police ^ • ^ 

11. Respect for law 

12. Support for constitutional rights ^ 

13. Attitudes to.ward policB - • ^ 

14. " Respect- for lavf ^'^ 

15. Attitudes toward pol*ice " * * 

^ 16. ' Support for constitutional xights 

5. Discuss the following critical question: What are the appro- 
priate and inappropriate uses of likert-type attitude surveys? The fol- 
lowing points should be made: 

— »Inappropriate Ihses : (a) Likert-type surveys are frequentlj^^ not a 
reliable way^ to determine attitudes or attitude change of individuals, 
(b) they should definitely not be used for grading students, and (c) if 
honest responses are desired-y students- should npt place* their name^ on 
the' survey form. 

— Appropriate uses : (a) Well-constructe^ and tested Likerf-type 
attitude surveys can be very efficient, direct, and reliable instruments 
for assessing attitude changes in groups* (e.g. , a class, all social 
studies classes, or some of the'sample of students), and (J)) this -inethod 
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is" especially effective for determining pre/posttest changes for the 

purpose of evaluating curriculum programs and planning for instruction. 

i ^ 
6. State that a good Likert , survey must also include well-written 

iiridividual Likert items. Distribute Handout L6, '^Evaluating LiHert-type 

Items." Gi) over the' first page, which provides a set of criteria or 

characteristics of well-written Likert items, explaining and illustrating 

eacfi,. of tbese criteria* Now direct participants to page 2. In their 

grcfups, participants should evaluate this set of 12 Likert items drawn 

from\^rious LRE instruments by applying the criteria to each item. Go, 
c 

over the correct answers with the* full group. For your convenience, 
possible answers are provided below: « 

1. ^ (uselof "can , be" makes the' statement ambiguous) 

2. ' 5 (statement is too generafl) 

' 3/ 5 

* -4. .3 (this item^does not present an attitude that differentiates ^ 
among groups) 
5' Qood itein ^ 

8. 5 . . - . ■ ^' • 

' lOv 2 ^ . . • . ■ 

' 11. 4 ^ . ^ * . 

. 12. Good item ^ ' . 

7 . State that a variety* of response formats\f or; Likert items ar.e 
possible. Distribute Kapdout J3|» "Response Formats for Attitude 'Scales / 
'and discuss the following joints': What formats are most Appropriate for'^ 
elementary and' secondary students? How many reappn^e choices are. best? 
Should a midpoint be.used?^ The following, pointy should be brought out ^ 
duiring the discussion: 

— Number of points ; Although there are no absolute* rules', the 
following* seem most approp'riate for ea'ch grade iQvel: primary grades — 

choices; intern^ediate and middle Schooi'-3-5* choices; secoYrdary — 
choices. Having mor,e response choices than suggest ei| here usually 
increasea^the difficulty for respondents without j^roducing more useful 
data. ^ \, . . 



• — Midpoint ; Again there is no absolute rule. Including a midpoint 
will allow some respondents to avoid thinking about Lheir opinions, but 

^the midpoint may accurately reflect some respondents' views. Not includ- 
ing a midpoint can arbitrarily force respondents to one side or' th^ other 
or can encourage them to leave items, blank. Inclusion of a midpoint may 
al3o have some effect on data analysis, especially as jt relates to*. 
interval and ordinal scales (see Activity 14 tor more information on 
1. this issue). Generally, use of midpoints is a matter of preference, but 
the point should be clearly defined so that the'results can be accurately 

^interpreted . 

. 8. State, that by applying the knowledge and skills gained from 

the previous activities, participants ^are now prepared to create their 

' % ' ' « 

own likert scales to measure student attitudes related to their LEE 

programs. Identify som^ of the LRE attitudes with which the workshop 
has already dealt. Ask participantsHo brainstorm other attitudes that 
might be important' to assess. This is done merely to stimulate partici- 
pants to think about Xhe types of attit-udes on which to focus* their 

, development efforts. ' , ' 

9. Ask participants to select a specific LRE attitude (e.g., 
attitude toward police), define several aspects of that attitude, d'efine 
a scale for that attitude, and create eight or ten items and a response » 
format which wil] measure that attitude in either elementary or secondary 
Students. Participants can do this activity individually or in small ' 

— .- groups depending on their preference. Circ-uiate to help participants 
-/* ' Jthrough the various steps and to answer questions. 

, ' I 10. Have several persons^ pr groups. then share the scales and one 
or two^tems. If time and materials are available, this can 'be done on 
* poster paper or on overl^ead transparency. If not, the sharing can be 
^ ddhe"^ orally . Con'clude by conducting k brief discussion of problems 



* *• ^ncountcre^ in developing itgms. 
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Handout 13 . ^ ' ^ 1 of 1 

(Activity 9) ' - 

JUVENILE?^JUSTICE CLASSROOM POLL ' 



After reading each of the statements below, mark the appropriate letter(s) in Ihe blank space which most ac- 
curately reflects your ppinion of the statement. There are no right orwrdng answers. 

SA*StronglyAgre0. A'4gr^e U-UncertMlh D-Dlsagre^ SQ-Strongly Dtsagr00 



_ 1 Police getj:riticized too often by young people for just trying to do their jobs honestly and fairly. 

_ 2. °arents should be held legally responsible for the actions of their children, whether in school or out. 

_ 3. In our justice procear&es^the distirv:tion between juveniles and a'dults works to the advantage of young 
people by shielding them from the brutal aspects of the system. 

_ 4 The existence of juvenile detention centers, reform schools, and prisons discourage or deter young 
people from comrnitting crimes. 

' *■ ' 

« 5. There should be k set penalty for each youthfiJI offense with little or no discretion left to the juvenife 
judge as to what the sentence should be. 

1 6. Shoplifting is just a matter of "ripping off" the establishment; therefore, shoplifters should not be pros- 
ecuted to th6 fullest extent of the law. . . ^ ^ 

I » 
^ 7. A juvenile should be given a trial by jury for any offense for which an adult criminal would be tried 

« 8. When you get rigf)t dowh to it, your chances of receiving justice are determined almost entirely by your 
age,.sex, color; arid wealth.. - ^ • * " 

^ 9: Students should be financially responsible for replacing damaged properly. 

JO Students should be required to "work off" any damage done to school property (example, cleaning desk 
tops, cleaning lavatpry walls, etc.) 

.11. Students should report to the office any individuals that tf]^ see committing an act of vandalism. 

J2- Students commit m6st acts of v,andalism because they want attention from their, friends? 

J 3. Most young peoplff*who get caught jn delrnquency really waht to be discovered and helped. 

J 4. Some element of punishment should be'a part of any gqpd co'rrectJons/rehabilitation program. 

.15. If parents and teachers would only be more strict, young people would grow. up respecting law and 
authority. ' " • 

.16. The personality and behavior of a child *is determined by the atmosphere within his home and the roles 
*his parents play in discipflne and setting standards of conduct. ' ' ' 



From the Institute fbr Political and Legal Education (Hf^E) , 
207 Delsea Drive, RD.4, Bpx 209, Sewell, New Jersey 08080/' Used by 
permission. • - " ' ' • ^ 



V, Handout lA 1 of 2 

(Actj^vity^9) 



POLITICAL ATTITUDE SCALE • , 

Directions ; The .attitlide survey on the next page is comprised of the 
four subscales explained below. Match the ' items.. on the survey to the ' 
subscales by placing^ the letter corresponding to the "appropriate 
subscale in th,e blank beside each item. - "r/ ' ' * i*-^ 

" Sense of Political Efficacy— feeling that ' the individual can 

^ ^ Influence political decision making. Individuals w^th a high 

sense of political efficacy believe that they have a voice in 
^> what the go,verninent does. They l>elieve that the government is 
responsive to their interests .and needs. 

-B.^ Political Trust — absence* of political cynicism. Political- 
trust means positive - supportive attitudes ajid belief s , about 
poUtical leaders and \he political, system. Individuals with 
- n high political trust believe that government officials are 
'^honest and competent. The}j believe tha^ public ofticial'S- 
« , ^^esire mainly to serve the people that th^ represent rather 
than themgelves. . . , *^ 

C. Equc^J.ltarianism — support for social or welfare democracy. An 
individual who is high in equalita;?ianism supports public 
policies .that would ameliorate or eliminate impoverished liv- 
ing conditio^ns. An- individual who is high in equalitarianism 
* supports action by 1?he government to equalize oppt)rtuhities 
for ind^ividuals to enhance their style of life. 

^ . Government^'l Support — willingness to favorably evaluate the , 

' - Presidency, the ' Congress,.^ and the Supreme Court. A ^person 
with a high score on this scale luas positive feelings about 
these institutions. He or she is supportive of thti.' present 
working of the Presidency , the Congress, and the Supreme ''Court . 



-Ad^pte^ from Howard Mehlinger and John Patrick, American Political 
Behavior (Bloomingjton , Indiana: High School Curriculum Center In Govern- 
ment, University . of Indiana, 1969). Used by permission. 



93 



4 

I • . 

Handout 14 • 2 of 2 



Key: A = Sense of Political Efficacy ^ C = Equalitarianism 

B = Political Trust J D = Governmental Support 



V 



^' The government ought to make silre that everyone has a 

i\ goo(i standard of living. 

-__ 2. Congress has done^a good job and should not be drastic- 
41 ' aliy changed. 

3' Almost all of the people' running the government are smart 

people#who usually, know what they ire doing. 



Public ^officials do npt car^rauch about what people like 
my parents think. * , " 



5. People like ray pferents can have an influence <upon the 
government by jdining groups of people (interest and 
'fjr^ssiire groups) that support the same political ideas. 

6. Money should lk)t be taken from richer people through 
taxes in order to provide se^rvices for poorer people* 

7. Congressc has, been too slow^in dealing with the majror ^ 
problems oi the United States. * 

8. The president has become too powerful and should have 
his^^fj^wers reduced. j ^ 

9. All people should have good houses, even if the govern- < 
raent has to build them. * ^ v 



10. 



Political parties are so big that the avevagec member has 
little or no influence on ^decision making. ' ^ 



11, Most of the people ruhning the government are a little 
cri)oked, . ^ ■ . • - -» 



12. ^ If poor people cannot afford to^.pay for health care, the 
government should pay ^their hospital and doctor bill«. 



13. What the government tdpes is like the weather: there* is 
nothing people can do about it, , 

14. There -is^alnio&t no connection 'between what a politician 
says and' what he or she will'dp ^when- el'ected-.> • 

. ^ ' . 

15. When the president of 'the'Unlted States makes asdecision, 
it 4s the duty of the people to obey him. 

People* in* the, government waste a lot of the money we pay 
ill ta^es, * » 



16, 
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Handout 15 ' ' ' 1 of 1 

(Activit:;' 9) ■ . # , . 



LRE ATTITUDE SURVEY 

1. If a person is suspected of a serious crime, the police should be 
allowed to hold him/her in jail udtil they can get enough ev.idcnqe ^ 
to officially charge him/her. 

2. Without rules many people would act less responsibly ^than • they do 
now . - 

3. At their trials, suspected criminals should have the right t© Refuse 
to answer question*; if they feel their answers may be used*against 
them. * / , 

A. Most police officers like to tell people what to do. 

^5/ Until they, are changed*, "even unfair *laws should be obeyed. 

6. Most police officers enjoy protecting and helping people. 

7. Students should be allowed to look at and chal lepgcL^their school 
recofds. . ^ - 

8. It is important to have laws which prevent innocent people from 
going to jail, even though some guilty persons may go-free* 

9. Local j>olice may sometimes be right in holding persons in jail 
. ' without^telling them of any formal charges against them. 

10. Police officers se]d«n make good neighbjDrs. 

11. Ignorance of the law is no excuse for violating it. , 

12. Emplo/ers should hire anyon^ who is qualified for the job, regarid- 
less of race, creed, or religion. 

13. Today, when police, ^question people, they use fair me^*ods. 
lA. It is all riglU: to break the law if you do not get caught. 

15. Most police officer^ treat all people alik^. 

16. * Some criminals are so bad that they should not be allowed to have a 
Jawyer. 
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Handout' 16 
(Activity 9) 



1 of 2 



YES 



NO 



EVALUATING LIKERT-TYPE ITEMS 
Guidelines 



1., Is the item phrised in words that are easily 
read by your students? 



2. Is the item stated clearly and simply? 



3. Does the statement express a clearly positive 
or clearly negative attitude?' 



4. , Is there only one dimension to the statement? 



5* Does the statement embody a characteristic* 

closely related to the attitude you are trying 
to measure? , 



Handout 16 * . ' . ^ • * 2 of 2 



A Potpourri of LRE Attitude Items 

Directions : Evaluate each item below using the guidelines given on the 
previous page. In the space beside each item, write "good" if you 
believe the item meets all thd criteria; if the item violates one or 
more of the guidelines, write the guideline number(s) in the space. 

Special courses about the legal system can be very help- 

0r ful to junior and $enior high school students in learn- 

ing about their government. (LRE attitudes) 



2. Police Irav^e Too many powers today. (Police author^^ty) 

1. 3:^ People should not be allowed to" march on public streets. 

(Political tolerance) 

' ^- ^Protecting everyone's rights Is. important. (Constitu- 
tional rights) 

5. Most lawyers are more interested in making money than in 

protecting people's rights. (Attitude toward legal com- • 
munity) 

' 6. " The right to privacy should be protected because"^ it is a 

constitutional' right and some police might .abuse the 
opportunity to make unlimited searches. (Privacy) 

7. Everyone should obey laws in order to avoid punishment 

. and to keep society functioning smoothly. (Respect fo;: 
r ' law) > # . 

8- " Police patrol my neighbo'^rhood carefully. (Respect for 

police) 

♦ 

9. Judges' protect our rights. (Constitutional rights) 

/ 10. Others- thoughts of one * s ^political actioqs are of no 

importance.. (Political tolerance) 



11. 



People have a right to engage in unusual or unpopular 
acts without cr:^ticism or punishment. (Constitutional 
rights) 



12. Women should, have the same righti- as men. (Equality 
before the law) 



97 



Handout 17 
(Ac!t:ivity'9) 



RESPONSE FOWIATS FOR ATTITUDE SCALES 
Elementary Level (Two-br Three Choices) i 



yot 1 



YES 
Like Me 
Agree- 




NO 



Unlike Me , ^ 

/ 

Disagree 



7 



0' 



YES 



NO 



Like Me ? Unlike Me 

Agree Don't Know Disagree 

■ © 





Secondary/ Level (Four or Five Choices) 



( 



Definitely Yes Probably Yes Probably No ^ Definitely No 

» <> 

T 

Strongly Agree Agree* Disagree Stjr'ongly Disagree 

* 

Hardly Everj - Occasionally Frequently Very Often 

Definitely Yes Probably Yes ' Not Sure Probably No Definitely No 

9 

Strongly Agree Agree • Uncertain Disagree StrongJy Disagree 
Almost daily 2-3 times a week 3-A times a month A few times a year Never 
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ACTIVITY 10 : , 
USING MULTIPLE-CHOICE TESTS TO 
I EVALUATE STUDENT LEARNING IN LRE 

Introduction : Multiple-choice tests are one of tha most frequently used 
methods of evaluating student — cogiiitive learning in LRE.* Weil- 
constructed multiple-choice tests can measure conceptual understanding, 
thinking skills, and factual knowledge. This activity is designe^Lt<> ' 
teach participants the basic principlej^gJL-Cons^trru'cting multiple-cho^ice 
test itera^^f^r^hesje^^urp^^s^ Participants first examine and evaluate 
Existing items and then d'e^velop their own items using the specified 
'criteria. 

t 

Objectives ^; , Participants will be able to — 

'1. Classify each item in a set' of jjiultiple-choice items as measur- 
ing cognitive knowledge, conceptual understanding, interpretation skills, 
or attitudes/values. ' • 

. 2. \ Evaluate a set* of multiple-choice items by applying a spepific 
set of criteria^ ' » 

3. Develop a set of multiple-choice items for their LRE ^programs 
using the principles demonstrated. 

Time:* 2- hours 

^faterials Needed: Handouts .18-21 ^ 



Procedure : ' ' ' ' ' 

. I. Introduce the activity by stating that multiple-dhoice tests 

can be a very useful and efficient* method of evaluating student cognitive 

learning in LRE. .State that while most,, tests me"a'«ure^ only fi^tual knowl*- 

edge, items *cai;i be developed to measure conceptual understanding rfnd ' 

interpretation .skills as well. E^tplain, and -Ibllustrate- the difference ^ 

between these three types pf learniilg* ^ • . ' . - , 

2. Distribute copies of "Handout J8 , "Classifying LftE Test Itemsy^* 

' V ' ' . V 

asking participants to complete the exercise as indica>tfed« in' the direc- 

^\ '* ^ I * * 

tions. Participants must classify a set of^ items using, the three cate*- . 
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gories defined in ste^_L_and^ the Ccltegpry iitt^tudes/values. This last 
category is used to illustrate that, tests designed to aTisess cognitive 
learning often inadvertently include.attitude items. 

3. Go over the participants* answers_jjtjain-4^ providing the 

answers given below for ^dis^-us^rtonT^ , ' , 

2. CK or CU* ' , 9. CU " - - 

3. CU ^ 10. CJJ 

4. AV H. CU ^ . , • 

5. IS ^ 12.^ CK 

6. AV , ' 13.' CU ' - 

7. AV . 14. AV ' • 



During the discussion, point out that* some item^i can be classified in 
two or three ways. One factor affecting the way in which items are ^ 

i 

Classificid is the nature of the LRE instructional program. If informa- 
tion is provided directly in the instructional program, an item that on 
the surface appears to be testing conceptual understanding may actually 
be testing only factual recall. Stress that some of the items are atti- 
tude items; unlike their cognitive counterparts, these items do not have 
' universa^ll'y agreed-upon correct answers. Such items do not^elo^g on a 
cognitive -test ; they should be separated and labeled as o.pinion or atti~ 
tude questions. Students should not fee graded on theif responses to 
these questions. ^ * ' 

4, "' Summarize the main differences between the three types of 
cognitive learning using Handout. 19, "txan\ples of Good Mul'tiplej Choice 

Items*." "^ ' • > ^ ' " y * . 

5. ' State that the most important criterion for- a good* test item 
is that It measures the*^ specif ic cognitive objective that it is int?end^d 
to measure. Stress the importance of clearly defining the particuiat'^ 
cognitive obj'ective, ^ (Th^s is similar to defining the specific attitude 
sQales in the previous set of activities on Likert,,surveys. > Askj/ar- 
ticipants to brainstprm a list of main LRE cogent aceas (e.g.,^ divil 
law, critndnal law, etc) and identify the main features or content in 

. each' area, . * 



6. Distribute copies of Handbut 20, "Evaluating Multiple-Choice 
Items. "^jExplain and illustrate each guideline. ' »r 

7. ^ Ask participants t;o work in groups of twQ,or thtee to complete 
the worl^sheet as described in the directions. Notu that the intended 
answer^ to the itens are provided on the worksheet, as are the areas of 
law t5|rbe tapped by "the itetn. This information is needed to apply the 
guidelines. ^ 



Go over the worksheet with the full g'roup,^ using the answers 
suggested below^ to "stimulate discussion: > 

4l. 4 (The foils are rruch tri^jfer than the correct' answer and c is 



^ too obviously wrong.) ^ . 

j 2. 2 (The stem does not give a clue to what is desired. better 
stem would be: "What is the main purpose of laws?") 
\t 3. • Good item , ' ' 

f A* 3 (Some may argue that a is more effective in some situations.) 
^ 5. 1 (The phrase "all bufwhich of the following" is awkward an^d . 

may confuse st^udents.^ . ' • 

• / 6 . Good item ^ • - ' 

%i ^' 3, 5, a*id 6 (The stem does not indicate what actions the son 
"d is debating between. Because of the vagueness of "the stem, b ' 

might be a possible answer. Thus, j^e item, is not ^^godd one 



to test for the abilirty to identify values in conflict.') 
,3 and 6 (This^ stem depends upion opinions aboufr treatment 
centers rather th^n knowledge about them.) 
flonH -i r pm . ^ 



I V °r \ 

9 . Good Item 

10. 'Good item ' ^ ^ 

\ 9t ^ Distribute copies of Handout 21, "Suggestions for Developing 
Multiple-Choice Tests" and use this to point out other key principles to 
keep in mind. J* r - 

10. ^ Have participants work in small groups to select a specific 
LRE content area and^develop for that area: two cognitive knowledge" 
items^^^two conceptual understanding items, and two interpretation skills 

• items . ' ' ■ , ^ - / , 

11. Have participants present their items x6 the - full ' group for 
review-, comments,^ and" suggestions for revision. 



12* Conclude by emphasizing the importance of creating tests that 
mea'sure ^conceptual understanding and interpiretation skills as well as 
factual knowledge. Also state that the principles participants fiave 
learned can also be applied in, ev-aluating and modifying existing test 



items. 



r 
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Handout TS . j ^ 

(Activity 10) . . ' 



V CLASSIFYING LRE TEST ITEMS 

- Directions ; ^ Read each LRE test item below and determine the kind 
of. student outcome it^ is primarily measuring. Write the appropriate 
abbreviation in the blank next to the itemi 

: CK = cognitive knowledge (recalling facts) 

CU = conceptual understanding (applying a concept) 

IS = interpreting skills (reading tables and graphs 
and leaking inferences from data) 

AV = attitudes or values (expressing an opinion or 
^ belief) 

SE = something, else (none of the above) 

1- When the U.S. Supreme Court hands down a decision: 

^ A. a lower court may overrule the decision. 

B. it establishes law for the entire country. 

C . it may not overrule a previous decision of t%e 
Supreme Court. ^ 

D. the decision is an examp~le of -legdrsla-t-iron-; — 7 



i 



Values are; ^ ^ 

A. the same the World over. 

B. different tilings to different .societies. 

C. determined by the Suprene Court.- 

D. based upon rules. 

♦ 

Rules made by school boards in the United Stat;^es most 
closely resemble those laws or rules made in: 

A. a monarchy. 

B. a dictatorship. 

C. a democracy* , - . 

D. an anarchy. 



^at 



Vandalism is an act tl^t shows: 

A. lack of concern for property. 

B. creat iye expression. 

C. you are acting against human rights. 

D. '^society needs to reevaluate its values. 
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*. selected' CHASACtERISTICS OF* Jir\'ZNtLE 
COURS:^ DISPOSITIONS 



Reason Referred 




Manner o£ Handling 




Race and Sex 




Q Males 
■B Females 



According to the above chart, most ofjfenses cominitt'ed by 
juveniles are: 

A, crimes that apply only to juveniles, 

B, crimes that apply to both adults and juveniles. 

C, crimes against property. 

D, crimes against the individual, ' 

^ J 

When involved in a disagreement", you should protect your 
rights by : ' 

A, .buying a weapon, 

B, • resisting unlawful arrest, 

C, hiring a lawyer, 

D'. paying an a/resting officer. 

Educational progra%is in prison: 

A. waste the .taxpayers * money. 

B, help prisoners adjust to free "society when they are 
released, 

C. are only necessary for high school dropouts, 

D, are only used in state penitentiaries. 



Wljich of the following statements best describes a search 
j^^arraftt? It is a: . o 

\* . ^ . , ^ 

A. \court order allowing a police officer to search all 
o?*--your home and possessions, - " V 

xourt order allowing a police ofeficer'to seafch 
your home for only certain -items. ' 

C. police order allowing a police officer to search 
your home or your possessions for a certain item. 

D* an of ficial *form that must be filled out after ^a 
search ^haS been made reporting what has been tound, 
during the search. ^ * 

IS 

The basi<: distinction .between crime and delinquency is 
the: 

A. motive for the acju- ~^ 
age of the offender. ' 

C. nature ''of the offense. 

D. -punishment"" given, t 

Which of the following is the 6est example of juHicial 
review operating in the federal government? " 

A; The Supreme' Court studies Alabama's new voting law 
and declares that it violates the constitutional s 
rights of blacks. 

B. The President reviews the records of two court- 
martialed soldiers and recommends that their life 
sentences be m^de eligible fpr parole in five ''years. 

£. Judge Smith of the Second Distrl>ct^ Court summarises 
the legal issues in the wail fraud case and sends 
the jury out to decide on its .verdict. 

D. ^Congress reviews the President's military defense* 
budget and cuts $2 million from it. 

* 

"I'm not going to take the witness stand," wl\ispered the 
defendant to his lawyer. "The prosecutor will)! twist* 

-ev^T7tlTlTig-^--say--t-o-^m^ Wbat_-legal_^ 

right' is ^the defendant exercising? - ' ' 

A. Right to counsel. 

B. Right to app^l. 

C. Privilege ^against ^self-incrimination-. 

D. Privilege of habeas corpus. * 

A bill was proposed on the floor of the House of *Repr,e- 
sentatives, sent to committee, and returned to the House 
^loor for a successful vote. What is the next- step this 
bill must go through in "order to bec'ome a law? 

A. It will be sent to the President for his signature. 

B. ^ It will be sent to the Senate for a vote. 

C. It will be reviewed by the Supreme Court. 

D. It will become a law without any further steps. ' 
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ITEM 'BOX . 


1. 


A newspaper 
his job. 


can 'pri^t its opinion as to how yell the -President is doing 
* i. • • 


2. ' 


A newspaper 


can make up an untrue story about someone and pri^nt^jLt-*- ^ — 


3. 


A newspaper 


may print only news ^ppr^v^-d^by^he government. 




13. 


Which examples giv^en in the rtem Bo^ show what freedom 


i 




of the, press means in the United States? 


) 




,A. Item 1 only. " 
B. Items 1 and 3. 

"Items 2 and 3. « 
,D. None of the items. 



l4. Laws are made for the health and safety of the people in 
a community. Should a law be made requiring all children 
under 10 years of age to be in bed by 9:00^ p.m.? 

^ A. No, because the law would be hard to enforce. 

B. No, because parents should decide. ^ ' , 

C. Yes, because ^children need more sleep th§n adults 
do to stay healthy. 

D. Yes, because lawmakers know what is best for people. 
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EXAMPLES OF GOOb MULTIPLE-CHOICE ItgMS 



Cognitive Knowledge (Recalling facts) 

Example ; Which of the follot^^ing is the -main function of the U.S. 



Suprjeme Court? 

A. Making the laws. 

B. Enfprcing the laws. 

C. Interpreting the ]^aws, 

D. -Changing the laws. 



Conceptual Understanding (Applying a*concept) 

' Example ; ' Which of the following is the best example of a civil 
case? ^ - r:'* 

A. A citizen sues the town council for damages result- 
ing from a fall on a rough street. 

B. Two men are arrested for selling marljuanar. 
' C. A businessman is fined for pot havJpg the proper 

license to do busine^^s. 
D. Parents are charged for failing to ^end their child 
' tto school. * 

J 

Interpreting Skills (Reading tables and graphs and making inferences 
from data) 

Example ; Use the following graph on recent presidential elections 
to answer these two^ quest ions (note the letters yefer to 
particular candidates). 

J 

o 

K o 



/ 
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Adapted f rofn Educational Re'^oirrces Center, American Governmetit 
Issues (Newjork;- Teachers 'College Press, 1981), -p. 18. Used by p^r- 
mission. All righps reserved. . » . . V 
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Exaniplres of Reading GrapR ; * 

Which candidate' recei-ved the highest percentage of vote? 

• B. ^'"'t ^ - ' ^ 

^ : ^ 'C, ■ T 

D. J ^ , . 

' Which of the following best describes the relationship 
betwe^en. experience/ability and percentage of vofe in 
these elections as depicted in this graph? » 

A. The greater the percentage of favorable comments 
about experience and ability that a candidate 
receives, the higher the percentage of the Vote 
he/she will get. 

B. The greater the percentage of favorable comments 
about experience and ability tfiat ' a candidate 
receives, the lower the percentage of vote he/she 
will get. 

; , C. Candidates with extremely large or extremely small 

% percentages of favorable comments ^ about their 
experience and ability receive low percentages of 
the vote. . 

\ D. There is no relationship between these two factors. 

\^ 

U 

Exan^ple of Making .Inferences from Data : 



What campaign 'strategy is suggested by the relationship 
shown on'^this graph? ^ 



A. /ko^'a/lot of TV ads. 

B. . / dej*5t|[ipre^ experience and ability. 

C. Get 'popple to see you as someone with good experi-' 
ence*. and' attility . ^ • 

D. - Focus on other factors tiesides experience .and 
r:">i^^^' . ability. 



,While the two previous test items assess ability ,tp read what 
* is on the graph, th^ latter item requires students to make/ a 
^ ^peasonable Ifeap (inference) from the data on the gr^pK. 
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YES 



• NO 



EVALUATING MULTIPLE-CHOICE ITEMS 
Guidelines 



1. Is the item stated clearly and simply? 

2. Does the stem make sense. by itself? 

3. 'Is the correct answer unquestionably correct? 

* , f 

A, Do, the wrong answers represent common or 
reasonable errors? 



5. 



's the item not too easy and not too dif ff~ 
cult? . ' - . ' 



6» Does the, item measure the knowledge or under- 
standing objective intended? 



A 
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A Collection of Multiple-Choice LRE Items 

Directions : The following LRE test items have been coHecte'd from a 
variety of sources. The intended correct answers are indic'ated by a 
circle." The phrase to the left, of the item indicates the knowledge area 
the item is designed to tap. Your ta§k is to evaluate each item using 
the criteria given on the previous handout. 



1. If no law^ existed, which of the following would likely 
^ ' occur? ' * , 

Need for A. People would probably not survive, 

laws • B. Governments would be stronger. 

C. Families would move closer together. 

^) Acts of violence and acts against property would 

take pl'ace for a while, but eventually people would 
^ make laws . ^ 

' * . 2. Kost laws were made: 

, to protect the things we believe in. 

Purpose of B. * to provide' for the national defenseJ 

laws • C. to punish criminals. 

/ D. to have a written record, so that future civiliza- 

tions will remember us, 

3. Jimmy, a devilish lad of five, saw that 80-year-old Mrs. 
Smith was about to sit in her lawn chair. He raced over 
^ and . pulled the* chair, away. Mrs. Smith fell to the 
ground, injuring her hip, "Mrs. Smith sues Jimmy *s 
father. Who is post likely to win the case? 

A. Jimmy's father — Jimmy did not intend ^to injure the 
old lady, only to pull her chair out. 
Civil law- - ' Mrs. Smith — Jimmy intended to pull the chair out 

and his family should pay for her injury. 
J C. Jimmy's father — parents are not legally responsible 
for. the actions of their children. 

D. Nobody — the court refuses to hear the case because 
Jimmy is only fiv'e years old. 

/ - ' 4. Which of the following actions would most , likely encour- 

age young people to observe -school rules? 

V A. Use physical punishment for violators. 

School rules B. ^ Ignore infractions of rules. 

C. Keep threatening what might happen if someone dis- 
obeys. 

Compliment students /or the observation of school 
rules. * 
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^P^tents/gwardians, by lav/, must provide their children 
with, all, but which one of the following: 



Family law 



Juvenile 
rights 



Value 
conf licts 



A. F&od. 

sjB, Clothe-s and shelter, 
C ^^edical attention. 
LoVe. 



A. 
B, 



Which of the following is - not a right, according to the 
cur.rent Juvenile Court Act? 

' ^ . . H 

The right to legal counsel. 

'Phe opportunity to introduce evidence and cross- 
examine witnesses. 
(6^ The right to" a jury trial. * 

^D^. The right not to be a witness against one's self. 

^An undercover agent promises to tell no one of his work. 
He'^.f inds out about a bank robbery, captures the crooks, 
but ^leaves rbefore the police arrive so as not to blow 
his cover. His son sees him leaving and think^s he is an 
escaping crook. His son must decide between: 



^7) ^ loyalty aqd responsibility. 

B* honasty ax\^ injustice. 

C. pride and shame. ^ 
•D. ""rules and regula-tioris. 



Which of the following best describes' a community treat- 
meht center? 



,,Just ice 
system 



An attempt at nehabilita^tion. 
A result of pri.son overflow. 
A'failing of the court system. 
»An expensive shot in the dark. 



Dynamic 
nature of 
law 



In 1896 the Supreme Court decided^ in ,the case of Plessy 
vs. Ferguson that separate facilities 'for white and black 
Americans did not violate the 14th AmendmBnf'' Sj;^^ long- as 
the facilities were equal. This^ d'ecision lenl^'legal 
support for segregated schools in AmericaT 

In 1954 the Supreme Court reversed the decision of 
Plessy vs. Ferguson in the case of Brown vsi the Board 
of Educat^ion . The new Supreme Court ruling said that 
separate facilities were inherently^ unequal. Segregated 
schools were, therefore, a violation of the 14th Amend- / 
ment and were unlawful. Howcould this complete change 
iq the Supreme Courtis decision best he explained? 



© 

'c. 

D. 



The Supreme Court justices misinterpreted ,the 14th 
Amendment in lj896. 

The 1954 decision of the Supreme Court reflected 
Social changes in the country. 

The laws of the United States had changed^ by 1954. 
The lawyers who argued the 1954 case were better at 
interpreting the meaning of the 14th Amendment. 



Mr.^ Johnson, a candida'te lEor city council, was driving 
up and down the streets in a residential area throwing 
rolled and banded one-page leaflets from his car onto 
the driveway or lawn of each home.. The leaflet encour- 
aged people to y^te for him in the coming election. 

Two police officers stopped Mr* Johnson and told him he 
would have to find some way to keep the leaflets from 
blowing or jisk being arrested for littering. Mr. John- 
son told the police they had no right to interfere wi^h 
his freedjDra to "distribute campaign literature in the way 
hej wanted and' continued to distribute bis^ leaflets. 
Wnat legal action could the police -of f-icers take? 

A. They could impound his car and get a court order tq^ 
stop Mr. Johnson from distributing any more 'campaign 

J literature. 

B. They could stop Mr. Johnson right then and make him 
pick up all of the leaflets he had already distrib-^ 
uted. ' ' • ^ - 
They could arrest Mr. Johnson for littering because ]f 
his right to free speech does not give him the right' 
to litter." ^ 

D. They could confiscate the rest 'of Mr. Johnson's 
literature and force him to clean up the mess he 
had made. 



Xl2 ' 



Handout 21 
^(Activity 10) 



I 



1 of 2 



SUGGESTIONS FOR DEVELOPING MULTl^LET-tHOj.aS -TESTS ^ 



Suggestion^ for gtatltig Problem (The Stem) 



->• 



1. The lead or stem of the item must present a single, cehtral ppbb?-/ 
lem*' Test for this by cov-ering the choices and noting' whether the' 
stem, standing alone*,^ is intelligib.ie. Every ttiultipleTxhoice item 
should be usable a^ a. free-response iteirv-*- — , • 

2. The stem should be stated simply and accurately and sliould contain 
all material relevant to its solvation. Errors occur here primarily 
because of omission of a statement of assumption, because the stem 
is stated ambiguously, or because it has complex sentence structure. 

3. In most cases the stem* or problem should contain only material rele-=- 
vant to its solution. For example, "Johnny goes to the hardware^ 
store tp^buy some nails.* 'If the ptice^of nails is ten cents a 
pound, wfiat' does' he pay for twelve ^pounds?" (Poor) "What is the 
price of twelve' ^unds of nails at ten cents a ^ound?*^' ^BefeCer) 

4. The scem is better stated in direct question or in direct statement 
form than in an incomp^lete statement ' form, since in '"the latter 

. inadvertent verbal clues may lead to th^ best choice. ^ ' ' 

5^ The problem should be stated in positive form whenever possible. 
Students often respond, to negative statements as- though ^they were 
positive. If the item cannot be stuted in'positive form, underlin- 
ing the negative words (e.g. , not and never ) can be helpful. 

6. Items requiring the student to express an opinion or value judgment 
should, in most cases, ask 'for the opinion of an authority specified 
in the stem rather than the student's own opinion. 

Suggestions for Developing Choices ^The Responses) ' * - 

1. The choices should be presented in logical order (e.g., numbers 
lowest to highest )1; ; • , ' , * ' 



5. 



The correct choice should 'be placed at random among choic6:§'^Tno 
fixed pattern) . ' ^ 

In elementary school, a minimum of three choices should be given; 
in high ..school, a minimum of four. 

The correct choice shoul^ b'e unquestionably riglit; most of the time, 
errors are made here by not having^ the answer complete enough. 

The suggested wrong choices should -represent. -errors cotnmohly made 
by Students in class discussion rather than general misconception-s. 
The wrong choices must be wrong tor a specific reason (too general, 
too reBtricted, or incomplete).^ . — 



Pandout '2? . • , . " -2 cff 2 

6, , The suggested choices .should be as brief a$ possible. Avoid "the 
necessity of measuring .r^^ding skills. • 



7. Except in' lower 'elementary grades, the .choices, should be le\;tered 
' • ajrid capital letters should *be used for ease of seoriiig, 

8. , Irrelevant . clues should direct the student away from the right 

answer if he or she> is unable to answer tl}e problem. They shquld 
never direc't^the^ student to' t;he right answer; This princ4ple'Ls of 
great importance because so mai\y clues. are given in multiple-choice 
items. The clues usually fa^ll in the following categories: . 

' ^'Wording in the stem ^nd the best* choice Js similar. 
, * .The key word in the stem is associated with key' word in choice 

Choices are from dffTerent or varied domains. ' 

Grammatical inconsistencies between stem and choices eliminate 

some choices. 
Not all choices 'are plausible. ^ 
Choices vary in length and complexity. ' 



ACTIVITY 11: " ^ ' ■ 

USING INTERVIEWS TO COLLECT 
^ • ^ STUDENT AND TEACHER DATA 

Inttoduqtion : Interviewirig* is a very useful 'technique for obtaining in- 
depth information itom a relatively small sample. It can be an effective 
,way to probe te.acher and student^ attitudes and perceptions in LRE proj- 
ects. It is also ^ good method for ♦^obtaining specific suggestions for 
revision of materials and activities. In order $,0 produce valid results, 
however , ^the interyiews must be planned and conducted systematically and 
thoughtfully. This activity, which involves the particij^nts in roTe 
plays, is designed to teach participants the skills of successful 

interviewing and tb point out problems that need to be anticipated and 

\ • . 

resolved. 

3- ^ 

Objectives : Pa,rticipants will fee able to — 

^ 1. Observe an interview, and identify several positive and negative 
behaviors on the part of the interviewer and/or interviewee. 

, 2. List several suggestions for ensuring that interviews are well 
developed and well_conducted. 

3 . Construct a short interview form and conduct a successful 
interview with it. ' ' 

) \ ] 

' time : 2-3 hours 

Materials Needed : Handouts 22-24,^ newsprint and marking pens or ^ chalk- 
board and chalk ^ 

) ^ 
Procedure t , ^ . ^ • 

1, Introduce the activ-ity by stating .that interyiews can be very 
useful ways to gather data on LRE project^* but they are not easy to 
develop and conduct successfully. St,at^ that this activity will focus 
on some common problems associated with interviews and^ays to alleviate 
them. Also tell them that by the end of the^ session they will have 
constructed and conducted an interview related to their LRE proj^e^ts. 



2* Distribute copies of Handout 22, "Interview Guide for - 
Teachers." Explain that this interview form will be used in a simulation 
of an'LRE interview. Emphasize that this is not meant to be -a model of 
a good form—on the contrary, it has some def^l^iite negative aspects; 
Explain that while the two facilitators simulate the interview, the par- 
ticipants' tasks are to try to write down the responses of the inter- 
viewee and to observe some positive and negative things that occur ift 
the interview* 

3. Conduct the simulated interview*. (If two facilitators are not 
available, ask for a volunt^er< f rom the participant group and brief., that 

person in advance.) Sometime during the role play make sure the^ follow- 

> " » 

ing things occur: ^ 

— The interviewer fails to probe to get the necessary in-f ormation. 

—The interviewer probes too much and le^ds the interviewee ito a 
response. . ' 

— The interviewer canoot wrdi|| down all the information provided on 

an item. 

— ^the interviewee rambles on one or two items. •^'^ 
— Th§- interviewee gives a vague response. 

A. Begin the debriefing by asking whether participants were able 
to accurately record the data obtained for each question. Distribute 
Handout 23, "Code Book for Interview. Guide for Teachers," explaining 
that it illustrates the detail in 'which one can break down possible 
res^nse's to open-ended items for tabulation and analysis. 

5* Ask the participants to identify things that happened during 
the interview that caused problems for them. Make sure the points in 
step 3 above are mentioned. 

6. Ask participants to identify positive and negative aspects of 

thfe interview form. 

7.. Distribute copies of Handout 2A, "Suggestions for Developing 
and Conducting Interviews/' Discuss each point using examples from the 
simulation whenever possible. 

8. Divide th^ participant;s into pairs and have each pair develop 
a two-page interview form related to some aspect of their LRE projects. 

9. Have each pair get together with another pair of participants 
for role pl|iys of ^ their interviews. One member of each pair acts as the 

' . . lit; 

' 50 . 



interviewer using their form, wlfile ^the second member is the interviewee 
for the other group's * interview. The two interviews should not be con-* 
.ducted simultaneously, so the two persons not directly involved^in each 
interview >can serve as observers, 

10, Debrief the role plays by asking participants to' identify any 
problems encountered during the^ interviews. Post and discuss these 
^items, referring to the list ol suggestions as needed, 

11. ^ Conclude the session by pointing out the advantage of using 
interviews for LRE projects (see the introduction to ttiis activity). 
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(Activity 11) 

" ^ . * * INTERVIEW GUIDE FOR TEACHERS^ 

Could you give be a brief history of ^he process. by which " 
V * law-related education became a part of the curriculum in your 

,v school? (INTERVIEWER: REPHRASE THIS AS SUITABLE; THE POINT 



(INTERVIEWER: PROBE ESPECIALLY FOR TEE FOLLOWING POINTS:) 
a) ^ho was conVacted initially? 



i 



b/ Whctrelse became involved? 



c) Who \/^re the decision maker&^in this process? 



/ 

/ 

/ 



r!c • • ^ -^^8 



IS 

V. 


TO GET A BRIEF RUNDOWN ON HOW LRE GOT INTO THE SCHOOL- 
hrhNU NO MURL IHAN J MINUTES 0^ THIS QUESTION. ) 


TRY 






-* 


11 


12 


\ ■ 




13 


1 / 

14 


' V 








* 




15 




— 


1 


"TT 


18 










f 


> 

/ ' '« 







19 20 



21 22 



23 24 



- '/( 25 26 

• , , - / r , ^ ■ 

/.s"i • • , - 

'/V.- ^ 27 28 

/.v:-^ . 

d) What factors iafluenced them? " 

* ' V — — — ' 



33 34 
"35 "36 



37 3§ 



39 40 
"4T"42 
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2. 



3, 



Have you ever had any training in .the law, 'other than in 
the sessions you have already mentioned? 



Yes 

No (SKIP TO # ) 



Can you tell me briefly about this training? 

(interviewer: ' find out i^ any of. the projects were in 
'any way involved in this training.) 



~3l 



33 



36 



To what extent' has ;your LRE training been helpful in 
assisting you to develop an effective program (or class 
presentations)? 



34 35 
17 ~38 

I 

39 



Not at all 
A little 
Moderately 
A lot 



40 



Uak your experience^ teaching law-related education 
changed your^ views of young persons in any way? 



Yes 
No 



Explain 



6, 



Has ^our _experien^ce 'Veaching law-related education made 

any difference in the way you deal with young persons or 

react to their behavior (either good or 'bad)-? 

> 

Yes ■ ■■ 

No 



20 



Explain 



25 



26 27 



28 79 
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CODJE BOOK FOR INTERVIEW GUIDE YOR ^TEACHERS 



-^1. ^ Could you give me a brief history of the process by which LRE 
became a part of the curriculum in your school? 



11 
12 
13 
14 
15 
16 
17 
18 
19 



Initial impetus to adopt LRE came from ; 
Teacher 

Principal/building administrator - 
District superintendent's office 
State dept. of education 
Regional social studies. specialist 
Educators ' professional organization 
An educator's spouse 



11 12 



(CODER: CHOOSE UP TO THREE OF THE REMAINING CODES 
—21 through 49~T0, ENTER IN COLUMS 13-18) 

Local educators first learned about LRE f'rom; 

21 Educators in a nearby school 

22 Educators in another district 

23 Educators in another state 

24 Local school board 

26 
27 
28 
29 

31 
32 

33 
34 
35 

39' 



■ibDnca"3r-iaw--gi?ettp 

Judge or justice person^ in the community 
Others in the community ' - 



13 14 



Flyer, brodwre, or letter from, an NIJJDP LRE project 

Conference or awareness session by an NIJJDP LRE 15 16 

project 

Training session by an NIJJDP LRE project 
Personal contact with staff of an NIJJDP LRE project 



Contact with staff or materials of a non-NIJJDP LR.E 
project » ^ 

Other 



17 18 



The initial content used for LRE: 



41 
42 

43 
44 
45 
46 
47 
48 



Wafe developed independently by a teacher 

JEvolved out of an existing course (e.g., consumer ed. , business 
law) . , - 

Was imported from another school or district 
Came from an NIJJDP LRE project ^ o 

^Came from a non-NIJJDP LRE project 
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Ir-b. Who else, became involved? 

(USE CODES- FROM PRECEDING PAGE) 



21 22 



Ic. Who were the decision makers in this process? 
(USE CODES, FROM PRECEDING PAGE) 



23 24 



25 26 



27 28 



4 

Id. What factors influenced them? c 

Do not code the following platitudes: 
''belief that LRE was worthwhile" 
"it had something 'to offer" 
"l/we/they lifced the idea" 
"they found LRE interesting"^ 



29 30 



31 32/- 
\ 

33^ VT 



Perceived needs : 

11 Delinquency problems in the community 

12 Student unrest/ student riots 

13 Antidote for ghetto concei^ts of right and wrong 

14 18-year-old vote requires 'added emphasis on rdlfeppnsibility 

15 Open-class st;^ructure in school requires added emphasis on 
responsibility 

t6 Improvement of school/parent relations 

17 Improvement of school/conununity relations ' *^ 

18 Perceived school needs — unspecified 

19 Perceived community needs — unspecified (code here if not 
specified as "school") ^ ^ 

Expedience : 



21 The curriculum had room for more electives ^ 

22 High-quality teachers were available 

23 Teacher(€) complained of not enough to do 

24 LRE seen as a device to build student interest in an existing 
course w 

Encouragement from within and outside the school: 

31 Recommendation by teachers ^to administrators 

32 Recommendation or favorable input from district 

33 Recommendation or favorable input from state educators 

34 Interest expressed by justfce persons in the community \ 

35 Interest expressed by others in the community ' 

36 Studettt enthusiasm/student interest 35 36 
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SUGGESTIONS FOR DEVELOPING AND ' 
CONDUCTING ^ INTERVIEWS 

Developing the Interview ForiDs 

Specify the information you want and why you want it. [ ^ 

Anticipate responses to open-en4ed items and begin to, develop categories 

of answers. , • ' ' 

K?eep the forms short and concise. " r ^ 

. ^ D , 

Conduct pilot interviews and revise the forms. 

Training for the Interviews ^ " * 

Xrai^i interviewers on the specific use of the forms* • ' 

Make cTlear to interviewers what 'information you want and why* ^* ^ 

Specify exactly where and how interviewers should probe. 



Con'ducting the Interviews 

Explain the purpose of the interview to interviewee* ^ • * 

Assure the interviewee of confidentiality. 

Give the interviewee the form before the session if posSible. 

Be sure to probe when necessary without leading the interviewee. 

If you do not finish, set up another, time immediat;ely . * 

Go over youir notes right aft,e^ the session and clarify and elaborate as 

^ needed. 5^ ' . ' 

Write summaries of the interviews as soon as*.possible. 



ACTIVITY 12: 
USING OBSERVATIONS TO EVALUATE ROLE PLAYS % 

Introduction ; Many law-related education programs use a wide variety of . 
instructional strategies. This activity is intended to help participants, 
consider using alternatives to paper-and-pencil tests to evaluate these 
strategies and to provide thdn with the skills needed to use structured 
observations to evaluate role plays. 



Objectives ; Participants will be liable to — 

1. Link instructional strategies commonly used in LRE classes to 
specific evaluation techniques. 

2. Construct an observational instrument for use with a role play, 

3. Use ^eir observational instrument to assess a role play. 

Time; 1-1^ hours • ^ 



Materials Needed : Handouts 25-27, newsprint and pen^ or chalkboard and 
cKaI|c, bag of powdered sugar 
cardBoard or, real television 



cKaI|c, bag of powdered sugar, sign's indicating locations in a house, and 



Procedures ; 

1. Introduce the activity 'by having participants brainstorm a 
list of instructional strategies commonly used in LRE classes.. Post 
these on 'newsprint or a chalkboard. 

2. Ask participants to work in groups of three or four, linking 
each of the 'strategies with several evaluation methods other than 
tea^cher-made or commercial tests. While such" tests are perfectly accept- 
able techniques, other methods are also appropriate and may be more 
congruent with some LRE ^itistructional strategies. As each group selectSv 
'evaluation methods , they shou Id also indicate the strejigths and weak^ 
nesses of each. (While participants are comjileting this task, you may 
want to prepare for the role play activity which follows by setting up 
the props, recruiting* three participants to take part in the role play, 
and giving them copies of Handout 25, "Directions for Role Play Partici- 
pants.") • 



3* ' For each instructional strategy listed, ask one group to indi- 
cate which evaluation technique they selected, as well as the strengths 
and weaknesses they identified. - 

A. Indicate that the next stdp in the activity focuses upon the 
application of one evaluative technique — observation — to role plays. 
Distribute copies of Handout 26, ''Search and Seizure Role Play,'' Ask 
participants what specific knowledge, sj:udent.s could teasonably be 
expected to demonstrate in this role play, and how they might demonstrate 
that* knowledge. List responses for eaph role. 

5. Distribute Handout 27, "Observation Form for Search and Seizure 
Role Play,"* explaining that il was developed using the process the par- 
ticipants just completed. • 

6. Conduct the role play, having workshop participants complete 
the observation form as they w&tch the role play. 

7. Debrief hy asking participants how they evaluated each role 
player according to the checklist. Review the process for construction 
of observation instruments, and indicate that observation is an excellent 
method for assessing skills and knowledge but is not as effective for 
assessing attitudes. 
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DIRECTIONS FOR ROLE PLAY PARTICIPANTS 

DETECTIvi HUGH BETTERLOOKOUT 

You need a search warrant so you can catch Loui with the "goods." 
Before you can get a warrant, you must establish probable cause. 
An anqnyraou3 telephone call is not enough. The .first thing you do 
is stake^out Loui/s place. During^ the three-day stakeout, you. 
observe 27 men and wom^n entering Loui's house with TV-sized ^boxes; 
all of this activity occurs between 3 and 6 a.m. You also observe 
15 persons leaving with TV-sized boxes or uncovered television sets 
This number includes one petson who drove up in a TV repair shop 
truck and left with ten sets.' 

After the stakeout, present your evidence to the judge, requesting 
a warrant to search Louies home, 001/2 Slippery Way. But you for- 
get to state specifically that you want to search for missing tele- 
visions. Then present the warrant to Loui and search his home 
completely. Search everywhere, including dresser drawers, under 
the sofa, and inside his shoes, where you will find a bag of 
cocaine. You also find 237 televisions, 128 of^'which have been 
reported missing and are traceable through identification numbers, 

JUDGE MARY GOOD OLDTIME 

• « 

^ You need to establish probable cause before you can issue a war- 
I rant. To do that, you first inquire about the source of Hugh's ^ 
oj-iginal information. That, source is insufficient to establish 
probable cause. ,.You then ask Hugh for additional evidence. If you 
do not believe the evidence he offers establishes probable cause, 
, ask whatever queartions you think are nece-ssary to determine whether 
probable cause exists. If you decide to issue a warrant, you need 
to specify the location to be searched and the approximate "time of 
day (not, between 10 p.m. and 6 a.m. unless you think Loui is danger- 
ous). You forget to specify object of this search. You also need 
to specify a .time limit for conduct c^f the search; yoy can give 
Hugh no more than 10 days. 

LIGHTFINGERED LOUI • , ^ . * . 

When Hugh arrives at your home, you ask to see .the warrant. When 
^you see it, you check the following: 

r 

1. Location to be searched. > * 

2. No mor^e than ten-day time limitation. * 

3. Reasonable hours of search. 

You forget to check whether the objective of the search is speci- 
fied. While Hi^gh conducts the search, be sure he con-fines his search to 
appropriate places. Protest when he looks^ in your shoes; the police 
have no reason to believe a TV might be found there. 
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SEARCH AND SEIZURE ROLE PLAY 




This role p/lay will provide students with the 
Opportunity Ito demonstrate knowledge of search warrant 
procedures, las specified for the three roles on the 
attached observation form. 



ROLES : 



SITUATION: 



'Detective Hugh Betterlookoat of the Woebegon^e City 

Police Department 
Lightf ingered Loui, alleged "fence" 
Judge Mary Good Cldtime« 

An anonymous telephone caller has just informed Hugh 
Betterlookout that Liglitf ingered Loui is fencing 
televisions. 



INSTRUCTIONS: 



Each participant is responsible for ensuiing that the 
law regarding search and seizure is followed; they 
should thus ask whatever questions and present whatever 
evidence is necessary to do that. Participants should 
remember that the observers cannot read minds: any 
knowledge they have about laws regarding search and 
seizure should be expressed out loud. ; - 

I 



«> # 
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- . ' pBSERVATION FORM FOR SEARCH AND SEIZURE^OLE PLAY 

In this role play, students shouW^ be able to demonstrate knowledge of 
; search and seizure procedures appropriate to their roles. Listed below 

are appropriate indicators of knowledge for each role/ If the student*, 

assuming a rple demonstrates a particular behavior ,^ place a check in the 
' D column next to ttiSt behavior. - If the student has the opportunity to 

'demonstrate that behavior but does not, place a check in the ND column. 

If the student does nor-have the opportunity to demonstrate a particular 

behavior (e.g., if the detective * shows Loui the warrant before Loui has 

the chance to ask to see it) ,^cb^ck the NOP column. 

DETECTIVE (STUDENT: ' ) ' D ND NOP 

1.. ^Gollects evidence to establish -probable cause. 

2. Presents probable cause^evidence to the judge. ^ 

3. Requests search warrant, specifying objective 

of search. ^ ' 

4. Presepts warrant prior to search. 

5. Searches^ in places appropriate to the 

objective of th&.seafch. ** , 



JUDGE (STUDENT: ; [ _) Q ND NOP 

\ . 1. Seeks to establish probable cause. , 

2. Sp>e'6ifies loc-ation or person to be searched. 



3. Specifies no more than ten-^ay time limit for 
search. ^ 

4. Specifies reasonable hours for search. 

5. Specifies objective of the sea^rch. ^ 

LOUI (STUDENT: ) D ND NOP 

^ 1, Requests- to see the warrant. 

2, Determines legality of the warrant by: 

a. checking location or person. . ^_ 

b. checking time limits. 

c. checking hour^ lj.mitations. , 

d. checking 'appropriateness of the search. ^ ^ 



' ,^ - ACTIVITY 13: / 

COLLECTING DATA AND ORCTANIZING ^' FILES 

Introduction : A well-designed evaluation plan and good instruments for 
collecting data will be of little use if the data collected are not safe 
guarded and organized in a way that makes retrieval easy. ' This activity 
provides a brief reading on organizing and maintaining ^ata file^s.^' 

Objectives : Participants will be able to— 

1. List some impo^rtant is'sues^ in establishliig a <i<!X^ management 
plan. ^ _ ' . " 

2. "Describe one mVfhad of organizing data files to facilitate 
d^^ta retrieval. . *' ^ ^ ^, . 



Time : " 15 'minutes 



y\- 



Materials Needed: Handout 28* 



Procedures : 



1. Introduce the activity by telling the parMcipants that the 
benefits of a good evaluation design and excellent, in^t rum^ii^s can be 
lost due to careless or ill-planned handling, of data, , Point out that ''a 
carefully planned data management system can help preyent loss of ^d^ata^ 
facilitate data retrie^^al anc^^ analysis, and ensure that all the data 
needed are ^collected?-*, - ^ 

2. Distribute copies of Handout 2J8, "Data Management," explaining 
that it provides some suggestions for org<aniVing data files. 

3. > After participants have^-Wad* the handout, spend a few minutes 
discussing its ke^ points and' their applicat^ion to partf cipants ' proj- 
Bcts. 
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»^ ""' ^ DATA. MANAGEMENT > 

This bandout is not "intendied as a ^complete guide to the art of data 
management. Rather, it pi:avides general suggestions for improving^ aata 
management, .along with detailed practical recommendations. 

The first step^^ln, establishing an effective data management plan is 
to design a workable\f iling system, pne useful *way for projects to file 
evidence is accord^ing to the 'sites at which- they are ope^rating. ^For 
example, if b project Is working at thre6 high schools, a data file 
should be created for .e^ch;. test results and other data from Roeky 
Mountain High would .then be filed under that school. 

^Witliln each- site file, documents should be organiz^ed according to 
j3(,foject goals and obj^ectives. A separate file should be set up tor each 
project goal at each site. General project goals may need to be further 
refined to indicate the implementation objectiyes that must be achieved 
in order for the project to accomplish a. particular * goal. 

Another useful strategy is creating a data log for eac^' site file. 
Each log sheet should be labeled clearly by site and should include 
space to enjter the fcrllowing inf or*mation~for eal:"h ddcufnenT .filed: 
document number, document title,**.4nd the research goals and bbjectives 
to which the document 'applies. - ^ ' , ^ - 

As documents are collected, bhey should be coded by site, *assigned 
consecutive numbefs, and related ^to a project goal or objective.- for 
example, a .document g^^r.h.^^coded Sky High, document 10> project goal 
5, ob j*ective. B . The document should then be entered on the Sky High log 
sheet and be filed according to the coding information. To ^f acilitate^« 
comparing, data across sites, it may sothetimes be hplpful to use 
color-coded instruments; for example , questionn^iires sent to judges^ 
might b§ printed on gre^en^ g^aj)^r^, J^t.hase^sen 

-^eirc% — Otc^sTon'ally ,~£r^ocument pertains to several goals and objectives. 
In this case, the original ' should be filed under one objective/ and 
Vduimny" sheets — sheets listing the title ""bf the document, its code 
number, and fflq location — should'be placed in the files for the other 
applicable goals and objectives.'^ Use of "dummy" sheets saves ^ 
duplicating expenses while alHowing rapid i^dentif ication all evidence 
bearing on- an issue, ^ ^ 

This system^ of fallijg aMows review of the progress toward, any goal 
at any site as we'll as identification ©f unfulfilled implementation 
objectives that may* be blocking progress toward broader goals. It is 
suggested that projects appoint* particular staff riiembers pl:lmary " 
respoi1||ibility (or individual sites . The assigned staf f should .be 
^responsible for developing^ comprehensive knowledge about the program at 
their sites. They should review all site documents as they? are 
collected and systematically review files in order to gain detailed 
knowledge 'about the sites and identify topics needing additional 
documentation. It may be useful* for projects to develop, or have site 
personnel develop, time l^nes for implementation of the program. 
Critical points on these time lines cpuld then be used as the points at 
vhich detailed analyses of site progress are conducted. 
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Because some of the data collected may be quite sensitive, document 
security and confidentiality must be maintained. Access ^o the fij.es 
should be limited to those who' need to work with the data. Any jpublic 
use of. the documents shoi6.d be carefully controlled to protect the 
identities of both individuals and sites. 



ACTIVITY 14: 
ANALYZING AND REPORTING DATA:' 
SELECTING THE APPROPRIATE DESCRIPTIVE STATISTICS* 

Introduction ; A wide variety of statistical techniques can be used to 
effectively describe evaluation results. Care must be taken in selecting 
techniques because different statistics emphasize different a'spects of 
the data and because statistics can easily^a^^ misuse^ fo imply false 
information. The purpose of this activity •is to give pat;;icipants an 
understanding of commonly encounter^ed types of data and a method for 
choosing the appropriate descriptive statistics for each... It^alsa gives 
them practice in actually presenting d^ta» ' ' 

Objectives :^ Participants will be able to-- ~ 

1. Identify the type of scale (nominal, ordinal, or interval) 

used for a given set of data. ^ . ^ o 

^. Describe* the characteristics of each of thes^e three seale 

types. ' 

3. Choose t'^e appropriate descriptive statistics to ^se \^hei^^^ven 

^^--scale type and a specific research question; ' \ * 

Time: p hours. , > - ^ 

Materials Needed : Handouts 29-32, nev/sprint and marking pens, and tape 
Procedure ; 

1. Introduce the activity by discussing the dilemma of having to 
4eal with two conflicting requircfments in pWs^nting evaluation findings: 
^(1) presenting data in a way that the*^ audience will, grasp in'^the time 

they are willing *to give to it and (2) presenting the data acci^rately, 
without masking its subt?leties and qualif icatit)n^, and without tKe dis- 
tort^ions which can^ accompany simplification and summarization of data. 

2. Point out that using the appropriate statistical techniques 
4:an help resolve this dilemma. Stress that choosing tfee proper statistic 



*Th-is activity was developed by Robe"ft B. Abeison. 
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requires care, however, because each , emphasizes a different aspect of 
the data and because they can be easily misused to imply false ipforma- 
tions* Tell participants that this activity will help them choose the 
, techniques to' use in different situations, in order to effectiv^y pre- 
sent' their evaluatdpn findings. • * 

3. ^ Distriout^ the fi^rst two pages of Handout 29, Ide^itifying 
Nominal, Ordinal, and Interval Data." Explain that the different type&n 
of data have different characterfstics; you must know which type you are 
dealing with to .choose the appropriate statistics. Make^ sure the par- 
ticipai^ts understand the ^,dif ferences be'^tween the threa types of scales 
and how*^o use the flowchart on the handout.* Then give them a few 

minutes, tp work in groups of two or three to idei>tify the types of data 

ft ♦ 
for each it^ on page 2 of the'handout. 

' 4. Distribute the last two pages of the handout. Discuss each 

item, making s^ire participants understand the correct answer, the^ 

reasons, and the commeatis*-^ — " 
^— '5^.,. Distribute H^dout 30, "Common Descriptive Statistics." 

Explain that descriptive statistic^ are used to answer five general types 
^of questions. For each general type of question, the chart shows some 

common" statisticaT techniques that can be used for each of the thre6 . 

types of data.. Note ^that any t.echhique listed ma^ also be used for any 

bigher^ level scale, but not for any lower level. 

- ^, 6. Divide the participants into gr9up6 of four or five. Give 
each group a supply of newsp*rinl: and ^a 'marking pen. Tell. th§ partici- 
pants that each gtoup will have the same tasTc to work on for 45 minutes^ 
Tell'^them not to look at what the other gi:oups are doing, so that you 
will "have a nuinber of independent solutions ^to the probleh. When t^je^ 
time is up, e^ph small group will have a chance to present its solution 
X.6 the" entire-^ group. , g . s 

7. Distribute Handout 31 ^ "Presentation to School Board." Give 
the groups 45 minutes to work on the task. ^ 

8. .'^Select a group and have ^ne member make a five-minute presenta- 
tioj/on their solution. Tap^" any visual , aids the group has const?ructed 
to the wall so they can be seen by everyone. Have ^as manj groups report 
on their "solutions as time allows. 
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^, "9. - Have the entire group discuss which techniques worked best, 
which not^ so well, and why. Discuss which contained the most information 
in the molfet ^underst^ndablp" way . Have' the participants point Out any 
technical etror$, such jas using techniques ina'ppropriate^ to the typo of 
dSta involve^,^ omlJttiag titles, etc/ ^ ^ 

•lO*^ Finally,^ pass Qut Handout 32, "Some Suggested techniques." 
Give the group a few minutes to look it over, and then discuss any ideas 
this packet contaiii^ that were not covered in the presentations. "Be 
sure to point out that theVe is ^no slagle correct approach to presentking 
<iata, as long "as the "presentation is both comprehensible and tjechnically 



correct . 
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IDENTIFYING NOMINAL, ORDINAL, AND INTERVAL DATA 



Threye types of data -^^re-^ -commonly encountered in evaluation ^ 
research: nominal, ordinal, and interval* Ea^ch of these types of data 
,has* unique Vhar^Qteristics and requires different statistical 
techniques. The definitions are: • - - " 



— Nominal: scales^ comp-r-i^d of a set of independent categories 

— '"'^wTth no intrinsic order. ^ Example: ethnic group. 
(No arithmetic d'peratio'ns may be done on nominal 
data, even if a number is assigned to each category; 
these numbers 's^-mply represent names of the 
categories and* are not "real" numbers*?)" 

— Ordinal:^ scales comprised of categories that have some order. 

Examples: grades (A,, B, C, \>\ aqd F) ; Likert 
scales. * arithmetic operations , such as 

' ^ computing^jSneSfte , may be done on ordinal data, since 

these operations require equal intervals between the 
points on the scald:. , For example, fhe difference in 
^ ( knowledge between a student scoring A and a student 

^ scoring B on an essay test is not necessarily the 

same as the difference in knowledge between a 
student scoring B and a student scoring C. Howeijer, 
sin'fce -the categories have an order, ordinal sca'le 
scores may be ranked, the medi£^n (the middle score) 
may be computed, and other order-dependent 
' statistics may be used.). 

— Interval: , ordinal^ scales where the intervals between 

categories (i.e., points dn the scale) are all equal 
(sometimes called "equal-interval" data). ^Example; 
f oarge. (Arithmetic operations may be performed on 

this type of scale.) 



The 
you have 



following -flowcl\art can be used to determine which type of d^ta 




You have 
NOMINAL DATA 



You liave 
ORDINAL DATA 



V 

You have 
INTERVAL DATA 
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Directions ; ^ Indicate whether each of the following types- of data 
is nominal, ordinal, or inter^val: - ' 

0 

1.' Answers to a survey question asking for the respondent's 

^ ' " occupation,' * ' * 

IQ scores* 



^.3.^^ Answers to an interview ^question asking whether the 

respondent's expectations for a workshop were met. TKe 



4. 



choices 


are: 




V i ; 


expectations rtet 




(2) 


undecided 




, (3) 


expectations not 


met 


Answers 


to the qirestion 


in 3 


are: 






' (0 


expectations met 




(2) 


had no expectations 


^ (3) 


undecided 




(4) 


expectations not 


me^t 




expectations not 


met , 




expectations 


c 


(6) 


other ' response 




^ (7) 


did not respond 





5, Number of times a student was absent from class during a, 

semester. ' 

6, Answers to a question asking whether students felt they 

learned more from an LRp class than from other classes. 
TJie choices are: 

(1) yes ^ ' ' 

. . (2) no ^ . . 

\ 7. Answers to a question asking stfudents to indicate their 

attitude t^oward s^ome issue, on a 9-point' scale, where 
1 = strongly in favor and 9 = strongly opposed. 

8. Answer's to the question in 7 above, except the choices 

are: 

(1) strongly in favor ' * . 

(2) in favor ' ' ' * , ' , 

(3) undecided 

(4) opt^osed 

(5) strongly opposed 

9. Data from item 7 above (9^point scale) ranked such that 

■ . the student with the most favorable attitude in the cla6s 

sets ^ rank of 1, the student with the next most^ 
favourable attitude gets a rank 6i 2,^ and so on. 

' 10. Scores on a classroom test consisting ^f 25* 

multiple-choice items measuring ' knowledge of LRE 
material, * - . 

^ U. Letter grades for a. course: A, B, C, D, ai;id F, 
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ANSWER GUIDE* 



r. Nominal > 'T^^ anjBwers dould be "teacher," ;^J'princfpal,'"*" 
"lawyer," etc., independent categories that*have no order even* i^ codfd; 
with numbers. The numbers 'are totally arbitrary names. and hafve no rekl 
numeric meaning. *' ** ^ '* • 

2. Interval . Some people* feel that IQ scores are really ordi|jaU, 
because there Xs no assurance that 'an increment of one IQ point always 
signifies the same increment in intelligence. However, most people feel 
that because -bf the way they are constructed, IQ tests' produce interval 
data. This ekaifple illustrates that the line between ordinal and 
interval data can be hazy and' o|5en to argument. 

- 3. Interval . This scale has only two intervals: one between the 
positive response -and undecidexi, and the oth^r between the negative 
response ^and undiecided. Since "undecided" ca'ti be regarded as a kind of 
"zero reference 'pioint and^since the' choices allow only discriminations 
between a positive and si negative response* (but not the* {{iagni.tude of the 
response) , we can consider the s^e of the two inte,rvals a^ equal*. As 
with IQ scores, however, a strictly conservative approach would consider 
this an ordinal scale, since we cannot be *absolut;ely sure the two 
intervals are exaqtly equal. 

'4. Nominal . Atiding the choices indicated' destroys any overall 
ordec to the categories. Furthermore, the choices cannot be arranged in 
a'ny 'meaningful order. ' In fact, the scale probably no. longer even has a 
single dimensio^i; normally there is no reasonable way to .arrange the 
categories of a multidimensional scale in ^ny meaningful order. 

5. Itite'rval . The numbers represent "real" quantities; it is 
proper to .^-p^rform all arithmetic operations on these data. ' 

6. Interval" . This example is a little -tricky. Oixe's first' 
impression i3 that this is an ordinal^ scale. Technically, however, "all 
dichotomies cai> be considered as interval data because they satisfy both 
criteria: ,,they have an order (one category is 'always more of 
something th^^n the other) , and (2) since they only have one interval, 
"all" intervals are equal. 

- - 7^^*^ . Ordinal . ^ The scale is defined so as to have an order. 
However, ^ith ntoe categories, it is highly unlikely that the resulting 
£ight intervals are 'r^a] ly. all equal; i.e., it is. highly unlikely that 
the nine points are exactly evenly placed along the^scale. 

(> 8. Ordinal or Interval . This one requires- $o1ne judgment. - As ' 
difecussed in item 7," with' nine categories this is most probably an 
ordinal sc£il6. In item ^3, however, the case was made for a . 
three-cate'gory Scale being , interval. A Likert scale of five" categories 
.is> therefore iSomewhere in between. A judgment** must be made as to' 
whether the categories are approximately equally spaced along th-e 
particular scale' being worked with. ^ 
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9. Interval , * Another tricky one. When you^rank aiv^ ordinal 
scale scores, you create an- ir^terval^cale , because ^is^ks represent 
"real" numbers. However, it becomes a scale pf ranks, so that any 
conclusions you draw pertain .to ' ranks , not to the original attitude 
scale. For example,, you can compute the itiean,* but, it is the mean rank, 
not necessarily the mean attitude. 

10. Ordinal or Int^^al . Such a scale is usually considered to be 
intefrval data because ,the scores represent actual numbers — numbers of 
correct responses. However, unless ^tftie focus is on ranks, as disQassed 
above, the real question is whether an increment of one^point anywhere 
on the scale always represents the same increment in- degree of 
knowledge. Therefore, a judgment is again necessary. If the te^t items 
are all approximately the same level of > difficulty , the scale is^ more 
like].x to be interval. But if the^ test consists of, for example, 24 
easy items and one very difficult item, the. difference between the 
scores of 24 and 25 clearly represents a larger diffei:ence in knowledge 
than that represented by the difference between the scores of 23 and 24. 
In that case, the scaile would be ordinals 

11. Ordinal. A very frequent misuse of statistics is to compute 
grade-point averages by assigning 4 to A, 3 to B, etc. Simply assigning 
T\umbers to ^ordinal ^categories does not create an interval scale. The 
extent of the distortion introduced by performing arithmetic operations 
on ordinal data is determined by the extent to which the intervals 
between the categories are unequal. Apparently, most people feel that i 
letter \rades are close enough to interval data that the convenience of 
treating them as such outweighs the distortions introduced- 
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COMMON DESCRIPTIVE STATISTICS 





_ ^ 




Type of Question 


* 


Type of Data 






Nominal 


Ordinal 


Interval 




GENERAL PICTURE:' What 
general picture of 
the group emerge's 
from the^data? 


k 111 1 1 ■ 1 . . 

Pie charts 
Bar graphs^ 
Frequency distribu- 
tions 


Grouped f reque'ncy 
distributions 


Frequency polygons 
Cumulative frequency 
graphs 

t 


• 


CENTRAL TENDENCY: 

What does the "typ- 
ical individual" 
look like? 


Mode ' 


Median 


Mean 

%. 




VARIABILITY: How homo- 
or heterogeneaus' is 
the group? * 


(special techniques) 


Range 

Interquartile range / 


Standard deviation 
Variance 


t 


INDIVIDUAL POSITION: 
How does some sin^ ^ 
file 'individual ( s'i 
compare to. t?he 
rest of the group? 


(special techniques)- 


Ranks 

Percentile ranks 


- ' 

Standard scores (a*g.^ 
z-scores) 




RELATIONSHIPS; How do 
specific variables 
— relate to each 
* other within the 
group? 


Contingency tables^ 
1 • 


Rank-order correla- 
tion 


Coefficient of cor- 
relation ' 



Note: 




Any techniques shown may also be used for any higher level scale-;, e.g. nominal .s^cale techniques 
majbe used with ordinal or interval 'data, but interval scale technigues may not be used with 
nominal or ordinal data. •* ' . 
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PRESENTATION TO SCHOOL BOARD 



The Task ' _ - 

You are a teacher at Grumpei: High School'. The principal has asked 

you to attend a school bo^rd meeting at which a recently implemented LRE 

program will be discussed. You are among the few teachers in the 

district who infused LRE into your U.S. history class for the first time 

this semester. Now that the semester is over, the school boai^d wants 
to know how things went. 



fi^ 



^During the discussion, you will be called upon to give a 
-minute presentation of^data to the boards Your task now, is to 



prepare one or mote visual aids to -help you present your data." « 
' Your Data 

* * ^* » 

You have two sets of data to present. One set comes from a test, 
consisting of five short essay questions concerning leg^l rights and 
responsibilities that you gave to your class at the end of the semester. 
You scored the test by grading each question as follows: 0 = wrong or 
no answer, 1 = partially correct, 2 = adequate answer, and 3 = excellent 
answer. Thus, students* scores could range from 0 to 15. 

Your other set of data comes from five Likert-type items you 
selected from an "Attitude Toward Participatory Citizenship" inventory 
1 and also gave to your class at the end of the semester. * On this 
attitude .inventory , 1 = strongly disagree, 2 = disagree, 3 = undecided, 
A = agree, and 5 = stron^y agree. The five items were: 

1. I can have a voice in what government does. 

2. If a law were passed which I believed was unjust, I would work 
to have it changecl .'^'^ 

3. I would be willing to' serve on a jury. 

%. Most people in government care about what people like me think. 
5. I would be willing to volunteer time to help a candidate with 
whom I agreed. 

You added up the ^responses to give an attitude score for each student 
ranging from 5 to 2^. ^ ^ 

You also a^ked a colleague who did not infuse LRE into a similar 
American history class to give the same tests, sd that you could have a • 
control group. The data for both classes are shown below. 

Some Hints ^ ^ 

All tables or graphs should have a good title and provide the 
-information necessary to interpret the d^ta presented. All columns on, a 
table should be titled and the axes of all graphs should be, labeled. 
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,.The^,inode is the most freqtfent score. The median, is ttie middle 
score {iTe,! half the group 'Scores above the median, and half scores;; 
below it). The meaix.is tie §jL|m. o^^the ^scores divided by the^number of 
scores. Note that in a skewed distribution, the mean is always more 
toward the tail of the distribution than the median, anji the median is 
always more central than the'mode (see diagram. below) • For this reason, v 
*^these .measures can ^give different impressions o.f the "typical ^ . 
individual," - . , " 



Number of 
Cases 

















Score 



Mo Mdn Mean 



If you combine data in a table, be sure you 'do not mask some ^ 
important characteristics of the group. For example, if a distribution 
is bimodal (has two modes),- don't combine the data, into categories in a 
way that hides this' bimodality • 



Always report the number in youf- sample. 



Data 







Essay. 


Attitude; 






Essay 


Your. 




Test' 


■ Inventory 


Control 




Test 


Students : 




Score 


Sco're 


Students: 




Score 


Student 


1: 


7 


9 


Student 


A: 


7 




2: 


14 


5e. 




B: 


10 




3: 


8 


11 




C: 


. 11 




4: 


8 


12 




D: 


-10 




5: 


15 


10 • 




E: 


. 13 




6: 


9 


20 




F: 


10 ■ 




7; 


10 


1-5 




G: 


9 




8: 


11 


10-. 


• • 


H: 


9 




9: 


11 


' 12 




I: 


12 




10! 


. 7 


* 

8 




j": 


13 




11 


12 


' 18 




K! 


? 




12 


13 


• a 




L! 


8 




13 


6 


14 




M 


7 . 




14 


: . 13 


9 




N 


12 




15 


: 15 


16 




0 


11 




16 


: 12 


15 




P 


: , 6 




17 


: 13 


20 




Q 


: 8 




18 


: 5 


7 




R 


: 5 




19 


:. 12 


16 




S 


: 4 




20 


: 14 


15 




T 


: 10 



Attitude 
Inventory 
Scor^ 

17 
•9 
10 

8 

6 

5 

14 
11 
10 
15 
, 9 
12 

9 

, 14 
13 
18 

7 . 
15 
13 



lERlC 
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SOME SUGGESTED TECHNIQUES 



No 'single api^roach to'analyzing and reporting data is always be'st. 
Many techniques can be used, as/long as they Ire, technically correct «arid 
^ comprehensible to. the audience Some suggested t^chnidues arc described 
belW: . . • , ^ , - 

1. First make frequency distributions of th^ raw data in d form 
• that allows you to see what you really 1iave, (The' original raw data aire 
rarely easily iriAiderstood in that^^orm.) The ^xa.in.ple..i.elow represents 
^ the ^ssay test data for yqur cFass. 



Score Frequency 



\5 4 : 


2 


lA 


2 


13 


' 3 


12 ' • *. 


3., 






11 


■2 


10 


1 






s ■ . ; 


2 


7 


2 ^ 






6 . 


1 


5 


1 



If 



V 

2. ♦ (Troup the raw data ff-equency distribution in a way that allows 
easier interpretation. A minimum of five and *a maximiim of nine 
categories usually works out well. Of course, the size of each category 
should be the same, except in special situations. Below, the essay t^st 
data are grouped into six categories, each two points wide. 



Frequency » Frequency 

Score (your class) (control class) 

IA-15 . A ^ 0 

12-13 6 A 

10-11 3 • 6 . , 

-8-9 ^ 3 ' 5 

6-7 . 3 • -3 

A-5 1 2 



142' 



'HandflMt 32 



BEST COPY A!/A!LaBLE 



2 Of 2 



3* Compute .any measures of central tendency, variability, etc. 
that you feelowoul|i bk useful. In the example, wi. would select the 
medianx» as the most^ accurate measure^ af central tendency for the essay 
tfests^'because * ^ , , 



The distribution for your^class is skewed, so the median gives 
a more accurate ,picture^of the "typical ind^ividual" than 
either the mode or the mean. ^ 



, b. We feel safer using the median rather than the mean, since we 
can't be sure we liave interval 'data! 

Note that it easy to figure the ipedian by finding the middle score on 
the raw data frequency" distribution (not th^. grouped frequency 
<^distribution) . In the example fxdm the. essay test data for your cliss^ 
th^\tenth score from the- bottom is 11 and the tenth score from th& top 
is r&. We split the»dif f erence and call^^the medianr-i4-.r5^'-(-For large 
grouf)s, there ar^e methods for^omputing the median from the grouped, 
distribution; see a statistics book.) , 

A. Prepare a t^^^le or graph to display the data in an attractive ' 
manner that also, provides the most information in an easily understood 
forma^jt. examples o/'two common alternatives: 



COKPARISON or GRUMPER H.S. LM-INFUSED 
ICAN HISTORY CLASSES: BNI>»or-SEMESTER 

ON tEGAL RIGHTS AND RESPONSIBILITIES. 

» t 


VS. NON-LRE-INFUSED AMER- 
SCORES ON AN ESSAY TEST 


Tett 
Scor« 


X of Stud«ncs 
(LRE Infused Class) 


X of Students 
(Non*LR£ Infused CUss) 


^ ' U-15 


20Z 




OZ 


12-13 


^ 30Z 




20Z 


lO-ll ^ 






30% 


8-9 






2SZ 


6-7 






15% 


4-5 


5Z 




10% 


. Total N 


* 20 




20 




11-5 




9.5 



COMPARISON 0^ CRIMPER H.S. LIE- INFUSED VS. NON-LRE-INFUSED AMER- 
ICAN HISTORY CLASSES: END-OF -SEMESTER SCORES ON AN ESSAY TEST 
ON LEGAL RIGHTS*AND RESPONSIBILITIES. . ^ 



Number of 
Students 



il 



UZH LRE 

►tN-20) 

I I "non-lre 

(N-20) 



4-5 6-7 



8-9 10-11 12- 
Score on Test 



13 14-15 
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Conimercialiy Ava i lab l^^Kg sou fees * 

^Th,e resources listed ^below could' form tl^e basi^of'a project 

^director^s or teachei;'s core personal library on program evaluation. 

v\ " - * 

These resources ^ere selected for inclusion because* ihey, are practical 

' ' *^ ' ' . ^* •» ' 

and 'easy' to uncJfer^t'ahd.'^ - - ^ ' - * 

Mager,. Robe.rt JF. , PreparinR.^ Instructional^ Ob jectivjes, . . Belmont, Calif.: 
. F^aron Pitman, 197S (2nd ed.). 

9 

Mager's little programmed book *on the vriting of clear, precise ^ 
instructional objectives is still one' of the best "how-to-do-it" guides' 
in the field. It takes the user through ^the objective-development 
process in a straightforward, "step-by-step fashion. Chapters^ areJ^"Objec- 
tives," "Why We. Care About Objectives," "The Qualities <if Meaningful 
ObjectjiJvies," "Iden1:ifying the TetTninal Behavior," "Further Defining the 
Terminal Behavior^" "Seating the Criterion^" and "Self-test." 



Morris, Lynn Eyons, Carpi FitzGibboits, ^and Marlefie Henerson. Program 
Evaluation Kit .- Beverly-^Hills, Calif.: Sage, 1978. 

6 / 
O * ^ ' 

This set of .eight easy-to-read paperbacks is intended to initiate 

the 4iovicef^ into the. practice of evaluation. The^ texts are sequentially . 

^arranged to provide import^ant information at each step in the evaluation 

process.- Titles are: Evaluatpi's Handbook , How to Deal with Goals and 

Objectives , How to Measure Atft^itudes, How to Measure Achievement, How to 
' — — ^ 

C alculate Statistics ^ and flow to Present an Evaluation Report . Thfe books 
can be purchased indi>>idually or as a §et (at a reduced .price) . 



Patton, Michael QAiinn* ^ Qualitative Evaluation Methods . Beverly Hills, 
Calif.: ^age. 1980. 



Patton li^s written one of the finest introductions to qualitative 
evaluation currently available^ The boo*k provides a rationale fo^Jj^ the 
approach, as. well as a phild!^ophical^ orientation. It also offers prac- 
tical suggestions in such areas as obserying, interviewing, collecting 
data in the fieljJ, and data- analysis. 



Rossi, Peter, Howard Freeman, and Sonia Wright. Evaluation; A .Syst6fflatic ^ 

^^^^ ^ 

. Approach . Beverly Hills, Calif.: Sage, t979. ^ . 

' Thi.s comprehensive guide to program evaluation includes chapters^ on 

the following to.^ics: ''Uses of Evaluation Research," "Program Require- 

ments"" f or Evaluation," VRes6arch for Program Planning," "Monitoring Pro- 

- gr^m Implementation,"" "A Framewo^^^ for Impact Assessment," "Rigarous 

Methods f^r 'Impact Assessment," "Approximate Methods for Impact Assess- 

ment," "Measuring Efficiency," and "The Context of Evaluation Research." 
* 

Tuckmati, Bruce Wayne. Evaluating Instructional Programs . Boston: Allyn 

and Bacon, 1979. 

» I* 

This is a general guide to designing and conducting both formative 

and summative evaluations of educational programs. The guid^e provides 

suggestions for each step in -the evaluation process, presenting case . 

studies of evaluations as examples,. 

* * 

TenBrink, Terry D. Evaluation: A Pr<actical Guide for Teachers . New 
York: McGraw-Hill, 1974. * 

This clearly written textbook provides not only spectfic guidance 
and practice for teachers but also general consideration of the cgntexts 
and purposes of educational evaluation. The focus is primarily on stu- ^ 
dent evaluation, but "Curriculum and instriftrional evaluation, components 
of the decisidn-making process, are also discussed. Evaluation is 
treated as informatioa feathering for decision- making, rather^ than as 



merely "testing." The three' major segments of the book are "Concepts 
Important to an Understanding of the Evaluation Process," "Steps, in the 
Evaluation Process," and "Procedures for Constructing or Selecting Spe- 
cif i,c Information-gathering Inst^ruments . " Examples are drawn from all 
subject areas. ' * , *^ • 

Practical easy- to-underst and sources on the use of s'tatistics are 
more difficult to find. The- two books listed below are unfortunately 
out of print. Because they are easily understood by the novice, however, 
they are well wo^th a trip to the local jlibrary. If neither is avail- 
able, consult ah introductory statistics text — or a statistician! 
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^ Franzblau, Abraham A Primer of Statistics for Non-Statisticiar\.s . 
r New' York: "llarcourt, Brace, and World. 1958., ^ 

This little paperback provides an excellent, nontechnical introduc- 
tion to statistics • focusing on copc^pts and rationale, rather than 

formulae. Quick and easy reading. 

{_ ft * . 

Ci 

Senders, Virginia L. Measurement ando, Sta-tistics . New York: Oxford 
University Press, 1958, 

This very readable statistics baok is intended for use. in an intro- 
ductory colJLege statistics coutse. The material is somewhat, technical 
and -comprehensive , but it ^is about as clearly presented and understand- 
^hle as is pos^'sible. 

0 

\ 

f 

Resources in the- ERIC System ^ , " 

The' resources below are available through the ERIC (Educat/ional 
Resources Infoilfetion Center) system. Each resource is identified by a 
six-digit accession number. Abstracts of and descriptive information 
'about all of these resources' are published irf a cumulative index, 
Resources in Education (RIE)*. This information is also accessible* 
through three* major on-li^ne computer searching systems: DIALOG, ORBIT, 
and fiRS. , ' . . ; 

Most of these documents-* are available for viewing in microfiche 
(MF) at libraries that subscribe to the ER*1^C collection. Microfiche 
copi^es of t*hese documents can also be purchased from the ERIC Document 
Reproduction Service (EDRS) . Paper Qopies of some documents can also be 
purchased fxpm EDRS. Information about the. availability of every docu- 
ment listed is 'included at the beginning of the annotation, along w^th a 
code indicating the prices for^ bott^^^icrof iche and paper cgpy. (The„ 
order form at the end of this section contains a key to the price code 
and other information about ordering co^pies from EDRS.) If a document 
' is v\ot available frohf EDRS,, the source is provided* 
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^all, Samuel, Evaluating Educational Programs > Princeton, J, : Educa 

. tional Testing Setviqfe^, 1979- ED 178 59-6?* EDRS price: MFOl plus 

^ * c * 

postago,," PC available only from publishei\ „ 



The activities of ttte Educational ' Testing Service (ETS) in 
evaluating educational programs are described. Program' evaluations are 
categorized as needs assessment, formative evaluation, or summative 
evaluation. Principled .used by ETS researchers in evaluating programs 
are described ^or each of the phases of "^evaluation: (1) making goals 
ex^plicit, (2) measuring ]grbgram impact, (3) working, in field settings, 
(4)^ analyzing the data, an'd'^(5) interpreting the results. 

Gates, Debra M. Program Evaluatipn and the Management of Change. Using 
the Individual as a Frame of Reference . Paper pi^esented ' at the 

> AdulV Education Association Co^nference , ED 208 221. EDRS 

pric§: MF01/PC02 plus postage. ^ . " 

^ ^ The Omaha Teacher^ Corps Project has developed a comprehensive eval- 
uation plan involving not only concrete aata pui 
well. Its evaluation program consists of seven levels of evaluation: 
input;s, activities, participation, reaction, learning, practice change, 
and end result. Six exhibits illustrate the logistical matters con- ^ 
cerned with data collection and documentation of each level. One com- 
ponent of the evaluation "ptocess is the Concerns-rBased Adoption Model 
(CBAM) , a model that uses the individuarl as its frame spf reference. It 
views the change process within formal organization's as entailing the 
individti^al's moving through several identifiable stages of concern about 
the innovation and eight levels of use of ^he innoviation. Levels .of use 
a'^re assessed using a validat^ecii f ocused-intervie\f process**'. 



urriculum Review Handboolc: Social Studies . Oklahoma City: Oklahoma 
Sta*te Qepartment. of Education, 1981. ED 208 536. EDRS^price: 
^MF01/PC02 plus postage. . ■ . 

This,^ handbook' s eight sections cqnslst primarily of open-ended 

questions- and checklists that allow, .for ^ flexibility and .adaptation" in 
• * 

'reviewing, analyzing , ^and evaluating a school's soci'al studies curric- 
ulum. The major section on' program evaluation contains a lengthy check- 



list that teachers can use^to determin^ the presence of elements that 
M Quld be .included in a equality social . studies , program. The areas 

----- " £, 

covered are program philos^ophy, organization and administration, 
instruction, materials and resources, ar^d evaluation. 

De Voss, Gary and others. A System for Documenting and Evaluating the , 
Experiences of Pre/lnservice Teachers. . Columbus? Ohio;^ Ohio State 
University, 1981. ED 211 500. EDRS price: MF01/PCd2 plus postage. 

Ohio Stat^ Univeissity ' s College of Education has created th^ Student 
Information Sys^.epi^j(fiIS:)' asva^means of receiyipg feedback and evaluating 
its^ teacher education programs. SIS is designed to monitor students' 
skill development at many 'stages and takes . into consideration the inter- 
related factors that shape teachers. This document presents the reasons 
for creating <he SI^S and the system's components. The instrumentation 
of the SIS , Including,* tables describing the SIS components and a cross 
seccion of a stage in professional development illustrating the compo- 
nents, is incj.uded, 

• • - ^ 0 

Cl 

Elementary Program Review Handbook and - Secondgf*ry Program Review Handbook . 
Sacraiqento^, Calif California State Department af* E;ducatdon, 1981. 
ED 212 080 ^and ED 212 081. EDRS price: MFOTplus pos^tage for each 
document. PC available only -f rom publisher. 

California's review process has be^n designed to judge the effects > 
of the school program on stuoents and staff and to identify^. opportij[nities 
for improving the program. Tn^ elementary handbook tontaiiis three chap-' 
*ters: one describes the progra'm review process, the second ^explains how 



to conduct a review and includes^, the' crp.teria used to** judge, and improve- 
programs, '"and the third describes how tp report findings. The s^econdary^ 
handbook ^describ^s thjee parts of \a review^ the effect of instruction ^ 
on students, the effect of* support qnr instruction, and the^ effect of the 
improvem^Rt process on support and instruction. For ^each,- tjie haiVdbocfk 

^ \ ' • \ . . \ > :\ 

provides criteria for judging qyalityi a^guiije for collecting inlorin^-. ' 
tion, and worksheets for preparing suggestions on progxam improvement. 
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Evaluatdpn Guide / Atlanta: Georgia State Department oiE Educatian, 1979. 
ED 181 566/ EDRS price: MF01/PC02 plus postage. PC also avail-. 
• A able' from publisher. 

This resource guide provides ,s^tep-by-st^.|) guidance to help sdhools 
evaluate* an adopted program. It offers an overview of the approaches 
most useful for evaluating a schoolwide or di^Jtrictwide prograni adoption. 
It emphasizes that= eyaluatipn should be addressed from the f irst*^ day of 
planning to provide^data for short-term and long-term cUcisior\-making. 
Topics covered include evaluation puj;pose^ goals and objefctives, assess- 
ment instruments , data requiremen^ts , data collect ion, data analysis , 
monitoring, ancl reporting. . * 

Gold, Robert S.'and others. Evaluation Studies and Change: A Review of 
the Literature . 1979. ED 212 645.* EDRS price: MFOl-plus post- 
age^. PC not available. 

This document reviews the literature related to' the use of evalua- 
t.ion results. Discussed are analysis of evaluatioti^ results /and tech- 
• nical methods 'for implementing suggestions^. Twenty-one pages of selected 
references are provided. ' < 

Haftsen, Desna Wallin. Program Evaluation—Staff/Faculty Development: 

' — ' ' ■ ? ] 

Providlng^a FrameworK .- 1981. ED 212,209.' E;DRS, pricef}' ' MFOl/pCOl 
plus postage. ' * ♦ ^ , 

> Basic considerations for evaluating a faculty^ development, program 
are considered. The framework consists of .establ i'shing* the parameters, 
designating the purpose, assessing the developmental stage of the pro- 
gram, c^etermining general methods of evaluation, and defining the cri- 
teria^ for suctess. Determining the purpose of the program evaluation' 
requires a considerable degree of knowledge- of the prograirft and objec- . 
tiyes, as well as the political and economic factors, c\ffecting any 
particular program. Before deciding the method of evaluation to be used, 
those involved should take a close look at the stage of the development 
program to Ve evaluated. ''Methods of program evaluation include t?he ^ 
historical-descriptive approach, the measurement-correlational m6del, 
quasi-experimental designs /the dev,elopmental-intensive model, the 
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"action-research approach, illuminative evaluation, and the consultative 
approach. « 

Jones, Leon.- Educational Evalua tion: ^Service or Menace. 1979. ED 201 

Si ^ ' 'ir- 

-706. EDRS price: ,MF0l/PC02 plu? postage. 

I 'This paper analyzes educational evaluation and assesses its value' 

to -the .e.ducational process^ The development of evaluation research and 

r practice is outlined ^nd formative and summative evaluation techniques 
are compared. The facets ol formative evaluation examined include : 
(1) its nature ^ahd, application, (2) how- it is conducted, (3) when it 
should be implemented^ (4) where it is conducted, and (5) .who implements 
•the process. A ^general analysis of evaluation with special attention to 
the possibilities for negative effects concludes the paper. 

Program-I^elated Eva'luat ion—Manual ind Staff Development Package , Los 
Angeles: Log Angeles Unified School District, 1980. ED 210 282. 
EDRS price: MF0l/PC0-5» plus. postage. ' \ « ^ 

This manual is (Resigned to accompany^ staff development, sessions in 
which research and evaluation personnel discuss l?:ey areas with c^dminis- 

r 

trators, evaluators in schools, and area office personnel* Forms and 

methods for assessment and reporting of pupils* progress in a local 

School's instructfon^l program are illustrated. Methods of establishing 

arx* evaluation committee and the 'functioning of it to review program 

implementation, quality, out^comes for students and staff,' and the match 

between buciget .and. expenditures are outlined; Specific forms are repro- 

duped, and directions for their use are included. * 

* » 

Raizen, Senta A. and Peter H, "Rossi, eds. Program Evaluation in ^Educa- 
' tion: When? How? To What Ends? ^ashington , D,C.: N'^tional Academy 
of Sc^iences-, 1981, ED 205 614. EDRS price: MF01/PC12 plus post-^ 
age, PC also available from Natio^nal ^Academy Press. • 

In response to a provision' of the Education Amendments of 1978" con- 
cerning evaluation practices and procedures," this report examines tour 
aspects of evaluation .irf^e^ucation , focusing on how funds allocated to 
evaluation can be spint more effectively and y^ield more useful resultfif^ 



On this ^asls, recommendations are made to both Congress and the Depart- 
ment of Education. These are presented in an opening summary and then 
discussed more fully in-each xRapter. The first chapter is designed as 
an introduction to the background and scop6* of the report. ^ A definition 
of evaluation is given in chapter 2, which also addresses congressional 
concern wit|i uniform methods and measures in the context of delineating 
different type3 of evaluation and their « appropriate use. Improvement of 
the quality of evaluation forms the substanc^of chapter 3. Discussion ^ 
in chapter 4 centers upon the utilization and dissemination of evaluation 
resu The final chapter makes recommendations *f or improved management 

and organization and presents implications for this derived from preced- 
ing chap'ters. * • 

Roberts, Sarah. Communicating Evaluation Results . Palo Alto, Calif.: 
American Institutes for Research, 1978.^. ^ED 181 3A5. ^ EDRS price: 
' MFOl plus postage. PC available only from publisher. 

' The goal of this' module is to help users develop the skills neede;^ 
to produce an effective evaluation rep'ort in terms of content, format, 
level of sophistication, accuracy, and organization. The module format 
consists of an overviewj goals, objectives, outJine, time sche,dule, 
glossary, readings, skill development activities, and bibliography.. A 
Coordinator's -Guide iitcluded has detailed instructions for presenting 
the module in a workshop setting as^^well as the facilitator's roles and 
functions, and the criteria used in 'assessing the participants \achie^(yew*i». 
ment of module objectives. 

Russell, Dale and others.. Developing a Workable Needs Assessment 
Process . Los' Angeles : Los Angeles County Superintendent of 
Schools, 1977. ED 179 565r EDRS' price: MFOl/PCOA plus postage^ 

This training guide was developed for inservice training on imple- 
menting the concepts arjd practices involved in needs assessment. Fifteen 
steps are desctibed: (1) determining the scope and possible outcomes of 
the program, (?) establishing decision, rules , (3) s^tating goals for each 
component, (4) assigning priorities to the goal^, .(5) conceptualizing 
the data collection prOcBas*,"*J-(6) obtaining measurement instruments. 



(7) ob.taining the supeif intendei)t * s approval, ^8) collecting, summarizing, 
and reporting the^data, (9) setting final performance standards, 
010) determinitig needs, (11) determining the priorities among needs, 
(12) determining the feasibility of the desired' program*, (13) identifying 
the pcJssible causes of existing weaknesses, (14) developing program 
obj-ectives and 'activities , and ( 15). reviewing the proposed programs. 
Exercises include selecting and ranking goals, selecting a needs assess- 
ment model, and questionnaire development. A bibliography is appended. 

Spirer, Janet E. Assessing Experiential Learning in Career Education . 
Vaper presented at the Annual Meeting of the American Educational 
Research Association, 1981. ED 199 273. EDRS price: MFOl/PCOl 
plus postage. 

Although this paper v f ocuses bn assessing experiential learning 
strategies in career education, many of the points discussed would also 
^ apply to other areas. The following six points should' be considered 
before an assessment program begins: (1) assumptions, (T) goals, 
(3) outcomes, (4) learning environment?* (5) ro^le of assessors, and- 
(6) focus. Two basic types of assessment are performance assessmen.t'* 
(assessing an experience as it odcurs) and q^utcome assessment (evaluat- 
ing the-. result br end product of learning). The same techniques may be. ' 

or both types. Asses*smeht techniques include direqt as sessiueilt-, 
Self -assessment , work sample, simulation, and pape'r-and-penci 1 tests. 
Before selecting one or a ^combination of assessment techniques, .triangu- 
lat ion (verification of t^e informa'tion obtaine^d) 'and types o^ program 
outcomes sh'^ld be considered. Other factors . such- as level of realism 
*de§ired, costs,, time available, and staff available must be considered 
when designing the assessment scheme. 

Stbn^, James^ C. and Raymond A. James, eds^ Qualitative Evaluation ^ 

Berkeley, Calif.; California Council on *the Education of Teachejrs , 
J98K ED 207 991. EDRS price: MF01/PC06 plus postage. 

^ "Qualitative "^evaluation" is the ^theme of this issue of the 
California Journal of Teacher Education , Ralph Tyler states that eval- 
uatibn is essentially descriptive and using numbers does not solve basic 
problems. Martha. Elin V^jrnazza examines ^the issue of objectivity in " 
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history and its implications for evaluation. Johp W. Ratcliffe discus- 
ses some recent paradigm shifts in scientific thought and presents some 
met^hodological guidelines for. evaluation that are^'congruent*" with thes?e 
shifts.' Mark SI. John distinguishes between seeing what is happening 
and one's ituage of what is happening. Other articles describe a variety 
of evaluation studies. 

■ • -S ' •■ 

Talraage, Harriet and Sue Pinzur Rasher, Quantif:^ing Qualitative Data: 
The Best of Both' Worlds .Paper p^esent.ed ^t the annual meeting of 
the American Educational Research Association, 1981. ED 204 396. 
EDRS price: MFOl/PCOl plus postage. 

An approach for merging quantitative-qualitative data in order to * 
enlarge the evaluator's perspective and provide an enriched data base 
for evaluating elusive evaluation problems in school settings is 
described. A replicable model for conducting mini-^case studies and 
analyzing within and across school data illustrates that quantitative 
'and - qualitative dafa are integrative and serve a confirmatory purpose. 
Both qualitative and quantitative data have inherent limitations, but 
integration of the twp will exten^the scope of the data base and 
generate new variables. Abandoning the view of quantitative and quali- 
tative data as dichotomous permits the development of synthesizing 
methods for discerning program effects in an organized, rational, scien- 
tific manner. * * . • , ' 



V/anous, Donna S. and Will-iam ^. Mehreris. Helping Teachers Use Informa- 
tion: The Data Box Approach . Washington, 'D.C. : National Council. . 
on Measurement in Education, 1981. ED 212 653. EDRS price: MFOl/ 
PCOl plus postage. PC also^ava^lable from publisher. 

Describing the findings and insights gained from a two-year research 
and development proj^^ct, this document focuses on the current measurement 
needs of teachers and t43e instructional processes for meeting those 
needs." The article is divided in^o four sregments: (I) a description of 
the literature in the field of e^cmcational measurement that guided the 
development of the project, (2) ^a description of the project, its goals, 
and components, (3) a discussion of the results from the field trials, 
and (4) a set of reflections aboqt* recommendations for the professional 



development of teachers in the content area ^of educational measurement/ 
The "Data Box^' is introduced and described as an ^instructional package 
having teachers investigate the use of assessment data in a variety of 
instructional decision-making situations. Jlajor measurement concepts 
teachers need to J^ow, and a thorough evalyation plan, are given, 

Williams, Paul L/and Jerry R, Moore, eds. Criterion-Referenced Testing 
for the Social Studies . Washington, D.C.: National Council for 
the Social Studies, 1980. ED 196 784. EDRS price: MFOl plus 
postage. PC available only from publisher. 

This bulletin discusses current issues and practices in social 
studies assessment, ways to improve testing, and methodologies to 
strengthen the validity, reliability, and value of tests. The primary 
purpose of the bulletin is to^bring the social .studies profession up to 
dat^ .about the issues and strategies involving criterion-referenced 
testing. The bulletin's six chapters examine such topics as inadequate 
treatment of assessment issues in social studies, characteristics^ of 
criterion-referenced tests and their applicability, the National Assess- 
ment, and procedures for ensuring -psychometric integrity for criterion- 
referenced tests. 
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APPENlflX A 

SAMPLE AC^ENDAS FOR EVALUATION WORKSHOPS 

• * • 

. As you begin planning your use of the activities presented ir^ this 
h'andbook, y.ou should keep twojjuestions in mind: , (1) What is your major 
evaluation goal? and (2)' Who will be responsible for each step in the 
evaluation process? For example, your goal may be^'to evaluate your pro- 
gram,^either for the purpose of improving the ^progr^m or in order to 
determine the program' s impact , Individuals - responsible for various 
steps iji the evaluation procedure may include teachers, administrators, 
and project staff. On the other hand, your goal may be to evaluate stu- 
dents. Student evaluation may have a number of purposes — to diagnose 
student aptitudes and skills,, to check student progress, to compare stu- 
dent kKOwledge and skills against a norjti or ag^insd .their knowledge and 
skills, prior to enrollment ,in the LRE class, or to determine whether the 
materials, content, and teaching strategies selected are achieving the 
desired student learning. The evaluators in this case might be all the 
teachers involved in the LRE program. ^ ^ ' / 

To assist you in thinking about how you might use the 'activities 
provided in tlie handbook, we have developed thr^e hypothetical situations 
and have designed a workjshop format to accompany each. These formats 
illustrate hdv you can analyze your own situation and design an appro- 
priate workshop agenda. 

Situation 1 

Yoij^^re the social studies coordinator in a medium-sized school 
district (3 high schools, 14 junior highs, and 28 elementary schools). |^ 
During the first semester of the past school year, you chaired a curric- 
ulum committee of seven teachers and two building administrators, which 
designed a law-related education program for grades 8 and 9. The com- 
mittee selected ten important concepts (e.g'. , justice, responsibility, 
privacy) and developed a two- to five-day lesson to accompany each con- 
cept. They then wrote a curriculum guide detailing how these concepts 
should be taught within the context of the eighth- and ninth-grade social 
studies curricula. A list of supplementary materials for teaching these 
concepts is included as an appendix to the guide. The guide was distrib- 
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uted to all eighth- and ninth-grade teachers. for use during the second 
semester. 

The coimnittee would no\i. like to know whether teachers are using the* 
guide and, ff sq, how well /they feel t\\e program is going. Thus, your/ 
goal is to evaluate the extent of program implementation and the succfess 
of the program once implemented. You may'^^^ant-.to do some student testing 
to find out if insDruct ional goals are being achieved, but such testing 
will be optional at this point. 

The committee members, whi? are relatively inexperienced in the area, 
will share the responsibility for conducting the evaluation. The com- 
mittee members can devote one day to receiving training in evaluation. 

Before you begin planning the workshop, you read pages 3-19 of the 
handbook, "A Model for Evaluating LRE Projects,*' and determine that the 
committee members should understand how to establish goals and objec- 
tives,, should have some knowledge about a variety of data-collection . 
techniques, should know how to conduct interviews with participating 
teachers, a'nd should understand what to do with the data once collected. 
Based on those priorities, you design the workshop format shown below. 



WORKSHOP AGENDA: SITUATION 1 



Activity 
Number 



Activity 
' Title 



Purpose of Activity 



limc Required 

Related 
Evaluation 
Workstxpp ^ Planning^ 



6 and 7 



Goals for LRE 

and 
Stating LRE 
Objectives 



Evaluation 
Techniques and 
Law-Related 
Education 
Questions 
and 

A Variety of 
Evaluatioa,^^ ^ 
Techniques 



16 learn whether your program 
goals and objectives meet the 
specified criteri«i. You may 
have to rewrite some of them, 
the first step ia modifying 
the new program. . 

To identify the acTvancages and 
disadvantages of various quali- 
tative and quantitative evalua- 
tion techniques and .to choose 
those which will be most appro- 
priate for your specific needs. 
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2 hours 



2-4 hours 



2 hours 



3-4 hours 
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Activity 
Number 



Activity 
Title 



Purpose of Activity 



Time Required 

Related 
Evaluation 
Workshop Planning* 



11 



Using Interviews To learn tJ\e techniques of 

to Collect interviewing and to practice 

Student and conducting an interview. 
Teacl\er Data 



3 hours 



1-2 hours 



13 



Collecting Data 
and Organizing 
Files 



To establish a process for 
collecting data and organiz- 
ing files. 



30 minutes 



*Does not include time for conducting teacher interviews and writing d report. 

t 

Situation 2 

Five lith-grade teachers in your district have been * teaching a 
curriculum entitled "Law: Teen-age Dilemmas" for the past three years. 
The course was developed by a few teachers based on their perceptions of 
student needs and community desires. The principal thinks the course 
should be dropped next year, even though enrollment is high. The 
teachers have come to you, the evaluation consultant for the district, 
for help in-demonstrating to the principal how ignorant iOth-graders are 
of < the law "sind how badly they need such a course. The teachers distrust 
standardized tests and want to develop their own instrumentation. They 
are willing to spend one evening a week during January and February 
planning the evaluation. You select the series .of activities shown 
below fpr use with the teachers. ^ 

WORKSHOP AGENDA: SITUATION 2 



Activity 
Number 



Time Required 



Activity 
Title 



Purpose of Activity * 



Workshop 



Related 
Evaluation 
Planning 



Evaluation Tech- To stimulate te^ers to think 
nitjues and Law* aboup the mo^t appropriate 
Related Education techniques for use in testing 
Quesjtions stu(}fint knowledge. ^ 



Indicators and 
Techniques 



To provide more in-depth 
information about the various 
evaluation techniques and the 
relative -advantages and dis- 
advantages of each. 



1 hour 



1 hour 



75 1^8 



\ - .1 



lime Required 



Activity 
Number * 



Activity 
Title 



Purpose of Activity 



Workshop 



Related 
Evaluation 
Planning 



9, 10, 
11, and 

12 ' 



13 



14 



Using Likert 
Scales to Eval- 
uate Student 
Attitudes 

Using Multiple- 
Choice Tests- to 



To learn how to develop and 
administer four types "of 
instruments; Likert scal'es, 
multiple-choice tests, inter- 
views, and observation formats. 
(Depending on the techniques^ 
selected by the teachers, you'' 



Evaluate Student, may be able to skip* one or 
Learning mor6 of these activities.) 

Using Interviews 
to Collect Teacher 
and Student Data 



Using Observations 
to Evaluate Role 
Plays 

Collecting Data 
and Organizing 
Files 



Analyzing and 
Reporting Data: 
Selecting the 
Appropriate 
Descriptive 
Statistics 



3 hours 



2 hours 



•To establish a process f^or 
organizing and storing data. 
(If you have an ongoing, p roc- - 
ess, you' may want 'to skip 
this step.) < 

To learn how to design a report 
for the principal. Covered will 
be what type of data to report 
inwhat way to accomplish a par- 
ticular goal--in this case, con- 
vincing the principal to continue 
LREf. 



2 hours 



1 hour 



15 minutes 



3 hours 



>-8 



6-8 hours 



4-6 hours 



4-6 hours 



8-12 hours 



Situation 3 

You are the head of a curriculum committee that is just beginning 
its work. You want to b6 sure that tha program your committee develops 
has enough detail so ' that it can be evaluated when the time comes. 
After talking with 'your school district's evaluator, you h'ave decided 
that you wl^H take your committee one step further and will include 
suggested indicators of success with your program goals and objectives. 
The-'evaluation training outlined below will be conducted as part of the 
committee's regular work. 
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WORKSHOP AGENDA: SITUATIOI^ 3 



Activity 
Number 



Activity 
Title 



Purpose of Activity 



Time Required 

Related 
Evaluation 
Workshop Planning 



The Evaluation 
Process in U^E 



To learn that curriculum devel- 30 minutes 
•opment is part of a larger proc- 
ess which includes evaluation. 



Goals for LRE 



^ learn the appropriate cri- 
teria for writirig goals which 
can be evaluated. 



1 hour 



Stating LRE To understand*' the difference 

Objectives between goals and objective's 

, and to leScn how to write, 

acceptable LRE objectives. 



Organizing and 
Refining LRE 
Goals and ' 
Object-ives 



To write goals and objectives 
for the curriculum. 



Indicators and To list indicatops of achieve- 
Techniques ment for Qach of the previously 

developed goals and objectives. 



1 hour 



2 hours 



2-3 hours 
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f m .APPENDIX B 

PROJ-ECT LIST 

The projects listed below have developed instruraentatiort for use^ in 
evaluating LRE programs. The projects ate listed in alphabetical order 
by state. Each project's name and address are giv^n along vith A fist 
of the types of instruments available. The types o£ instruments \are 
cfoded according to ^he* following key:. 

INSTRUMENTS FOR EVALUATING CLASSROOM -INSTRUCTION/IMPACT ON YOUTH 

1 — paper-and-pfencil tests of student knowledge/skills 

2 — paper-and-pencil purveys of student attitudes/behaviors / * 

3 — student interviews ? ' ' ' * * 
♦4 — teacher interviews 

5 — teacher questionnaires . . ■ ^ - 

6 — classroom observation formats 
INSTRUMENTS FOR EVALUATING TRACING: 

7 — ^jg^[)er-and-pencil tests of trainee knowledge/skills 

8 — paper-and-pencil surveys of trainee attitudes 

9 — trainee interviews 

10 — trainer interviews - ^* ' ^ 

11 — training observation formats \ * • 
OTHER? INSTRUMENTS , ^ ^. . 

^ 12 — surveys/questionnaires ~ ^ 

13 — interviews 

14 — other ' 

All of the projects have agreed, to make 'single copie$« of thieirv 
instruments available to projects or progra"ms hav^ing ^similar evaluation 
needs. An asterisk next to a project's name indicates that a charge for 
copying will be^made. * 

This list is by no means exhaustive^ There are^ literally hundreds 
T.Pr [»v > J' ''» M ' " n^*"-^^^^* TTn;^^pH gf^feft today^ A nearly complete 

listing is provided in ABA's Directory ^of Law-Related Education Projects 
Those seeking particular-, kinds*' of evaluation instruments may want -to 
consult the Directory s project descriptions to locate projects with ^ 
objectives similar to their own. In^ fact, the Directory may also b^., • 
useful in determining whether instruments from the projects li-sted below 
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would be usfefuJ. to another project or program. The Directory ^ s fourth 

* » * ■ 

edition (1982) is available for $4.95 fVom ABA, 1155 E. 60th^Street; 
Chicago, XL 60637. ' . v ' ' ^ 



California 



District of. Columbia 



Curriculum Strategies for 

Delinquency Prevention 
ATTN: Norma Wright 
Santa Clara County Office of 
^ Education^ 

* iOO Skyport Drive 
San Jose, CA 95115 
Instruments: 2 (grade 12) 

'. Law ill s Free Society/Center for * 

Civic Education • ^ , 

. 51l5^.Douglas Fir Drive, Suite 1 
Calaba6as, CA 91302 
' Instruments: 4, 5% 6% 8, 9 • ^ 
« 

^La'W-Related Education Curriculum — ' 

Long Beach Unified School Dist. 
701* Locust Avenue 
Long Beach, CA 90813 

* Instruments: I (junior/senior 

high), 5, 12 (student:s) 

V 

Youth Action Center for Positive 

Change \ 
A39 W. Compton Blvd, ^ 
Compton, CA . 90220 
Instruments: 1, 2, 3, 4, 5, 7, 8, 

9, 10; school and nonschool youth 

are covered 

Colorado * 

. *LQW/-Re>lated Education Evaluation^ 
Projec?^' 
Box 3578 ^ 
Boulder, CO 80303 . ^ 
Instruments:, 1 (grades^5,.. 7-12), 
. . 2 (grades*5., 7-12), 4, 5", 6, '7? 
"* 8, 9, 10, 11 , 12. (law school 
dedns, community resource per-? 
sons, scfiool administrators), 13 
(school administrators,'^ law stu- 
dents, community fesource per- .x^ 
sons); school and nonschool* youth 
are covered 



^National Street Law Institute 
605 G Stfreet, N.W. 
Wa^hingtoji,'' DC 20011 ' 
Instruments: 1 (grades 10-12) 

Georgia 

^Georgia Center for Citizenship 

and Law-Related Education 
Georgia State University 
-Box 604 

University Plaza , 
Atlanta, GA 30303 
^Instruments: 5 , 8 , 9 , 11 

Improving Citizenship Education 
ATTN.^Helen W. Richardson 
Fulton County School System 
786 Cleveland Avenue, S.W. 
Atlanta, GA 30315 
Instruments: 1 (grades 1-12), 
2 (grades 1-12) 

Illinois L ' ^ ^ 

w ^ 

American Bar Association 
Special Committee <t)n Youth 4 
'Education fdr Citizenship 
^ri55 E. 60th* Street 
'Chicago, IL 60637 
Instruments; 8," 12 (secondary 
teachers) 

Constitutional Rights Foundation/ 

Chicago 
122 S.' Michigan, Suite 1854 
Chicago, "iL 60603 
Instruments: 1 (grades 9-12), 

2 (grades 9-12), 3 (grades 9-12) 
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Massachusetts 

Courts and the Classroom 
.District Court Department of the * 

Trial Court of Massachusetts , 
209 Essex Street 
Salem, MA 01970 

Instruments: 8, 12 (court person- 
nel)^ 

Michigan . 

School Justice Project 
Oakland Schools * , 

-2100 Pontiac Lake Koad 
Pontlac, MI 48054 
Instruments: , 1 , *j2 

Missouri 

*Law and -Education Project 
4130 'E. Lexington 
St. Louis, MO 63115 
Instruments: I (gjades 5-6) 

*Legal Rights and Responsibilities" 

'in a Free Society 
ATTN: Carl C. Fehrle 
31)9 Education Building 
University *o£ Missouri - ^ , . 
Columbia,- MOa, 65211 
Instruments: Sr, . 

Rights and R^ipspoiisibilities of 

Citizenship ixj a Fr^^e Society 
Missouri Baii^ \ 
Box 119 ' ' . 
Jeffferson City, MO 65102' 
Instruni^ts^ 1 (high school) , 7 

New Jersey ^ 

* *New Jersey Law^Related Education 
Project 

Inst^itute for Political and Legal 

. Educ^ation • * 

207 Delse^ Drive 
RD 4, Bo>: ^205 
Sewen, NJ 08080 
Instruments:- 1^ (grades 9-^12), 2 

(grades 9-12)^-3, 4, 5, 6, 9, 12. 

(potential pr-Dgram adopters) * . 



Wesf Orange Law Education Project 
22 Municipal Plaza * 
West Orange, NJ 07052 
Jnstruments: 2 (grades K-9) , & 

Oregon ' . 

Oregon Law-Reldfted Education 

Project 
220 SE 102nd 
Portland, OR 97216 
Instruments: 1, 7, 8, 12 (teachers)* 

/" 

Pennsylvania ^ 

^Community Involvement » for 

Responsible Cit izen'ship 
Two Allegheny C^enter, Suite 1300 
Pittsburgh, PA ^15212 
Instruments: 1 (grades 1-8), 2 

(grades 1-8), 4,5 

Justice Educatiori^l'eaching 
Strategies 

Pennsylvania Department oX 
Education 

333 Market Street 

Harrisburg, ^A 17108 

Instruments: 1 (grades K-6) , 4,' 
13 (parents, school administra- 
tors) 

South Carolina 

Ways of the Law 
2712 Millwood Avenue 
Colutnbia, SC 29250 
Instruments: 1 (grades 9-12), 2 
(grades 9-12) 

Tennessee 

^Tennessee LRE 

Peabody Center for Econfemic and 

Social Studies^* Education 
Vanderbilt University ^ . 
Box 320 ■ 

Nashville, TN 37203 
Instruinents : 6, 8,* ^9 
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Vermont 

Law-Relatei Education Project 
Department o^f Education 
120 State Street 
JMontpelier, VT 05602 
Instruments: 1, 5, 7, 12 



Virginia *' / 

VirgiTiia Institute £or Law and 

Citizenship Studies^ 
Schopl of Education 
Virginia Commonwealth University 
1015 'W. Main Stueet 
Rtchmond, VA. 23-284\ 
Instruments:' 1 (grades- 10-12) , 2 

(grades 10-12), 7, 8 



